[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:55 dist-test-slave-whjp krb5kdc[10687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:55 dist-test-slave-whjp krb5kdc[10687](info): set up 2 sockets
Sep 26 16:36:55 dist-test-slave-whjp krb5kdc[10687](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 16:36:56.358323 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:36:56 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:36:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:36:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:36:56.521449 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:45361
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:56.836298 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:56.836597 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:56.836710 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:56.843928 10700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:36:56.844039 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:56.844117 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:56.844187 10700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:36:56.844245 10700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:36:56.854255 10700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:56.855741 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:56.857041 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:56.868613 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:56.868785 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:56.868613 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:56.869292 10700 server_base.cc:1034] running on GCE node
I20260926 16:36:56.870074 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:56.871471 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:56.872668 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790440616872609 us; error 49 us; skew 500 ppm
I20260926 16:36:56.872990 10700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:56 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440616, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:56.878645 10700 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:36:56.880290 10700 webserver.cc:460] Webserver started at http://127.10.110.62:33355/ using document root <none> and password file <none>
I20260926 16:36:56.880980 10700 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:56.881089 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:56.881368 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:56.884308 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "41b1e853422246e9bb06f5c867029e12"
format_stamp: "Formatted at 2026-09-26 16:36:56 on dist-test-slave-whjp"
server_key: "fde83aeed25881eac8854ebd0661a6b7"
server_key_iv: "760cf7e5c34d6d061815167e31a82c2f"
server_key_version: "encryptionkey@0"
I20260926 16:36:56.885054 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "41b1e853422246e9bb06f5c867029e12"
format_stamp: "Formatted at 2026-09-26 16:36:56 on dist-test-slave-whjp"
server_key: "fde83aeed25881eac8854ebd0661a6b7"
server_key_iv: "760cf7e5c34d6d061815167e31a82c2f"
server_key_version: "encryptionkey@0"
I20260926 16:36:56.890081 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:36:56.893368 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:56.894651 10700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:36:56.894822 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "41b1e853422246e9bb06f5c867029e12"
format_stamp: "Formatted at 2026-09-26 16:36:56 on dist-test-slave-whjp"
server_key: "fde83aeed25881eac8854ebd0661a6b7"
server_key_iv: "760cf7e5c34d6d061815167e31a82c2f"
server_key_version: "encryptionkey@0"
I20260926 16:36:56.895102 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:56.922703 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:56.926908 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:56.927186 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:56.944247 10700 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:45361
I20260926 16:36:56.944255 10766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:45361 every 8 connection(s)
I20260926 16:36:56.945601 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:36:56.947113 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 10700
I20260926 16:36:56.947386 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:36:56.949222 10680 external_mini_cluster.cc:1428] Setting key d7c210c4f872abc0e2af64972c4b8c9d
I20260926 16:36:56.949404 10767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:36:56.959991 10767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12: Bootstrap starting.
I20260926 16:36:56.963635 10767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12: Neither blocks nor log segments found. Creating new log.
I20260926 16:36:56.964989 10767 log.cc:824] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12: Log is configured to *not* fsync() on all Append() calls
I20260926 16:36:56.968212 10767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12: No bootstrap required, opened a new log
I20260926 16:36:56.975503 10767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b1e853422246e9bb06f5c867029e12" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45361 } }
I20260926 16:36:56.975925 10767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:36:56.976022 10767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b1e853422246e9bb06f5c867029e12, State: Initialized, Role: FOLLOWER
I20260926 16:36:56.976689 10767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b1e853422246e9bb06f5c867029e12" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45361 } }
I20260926 16:36:56.976897 10767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:36:56.977022 10767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:36:56.977174 10767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 16:36:56 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:36:56.979423 10767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b1e853422246e9bb06f5c867029e12" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45361 } }
I20260926 16:36:56.980023 10767 leader_election.cc:302] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41b1e853422246e9bb06f5c867029e12; no voters:
I20260926 16:36:56.980661 10767 leader_election.cc:288] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:36:56.980784 10772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:36:56.981359 10772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [term 1 LEADER]: Becoming Leader. State: Replica: 41b1e853422246e9bb06f5c867029e12, State: Running, Role: LEADER
I20260926 16:36:56.981923 10772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b1e853422246e9bb06f5c867029e12" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45361 } }
I20260926 16:36:56.984045 10773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b1e853422246e9bb06f5c867029e12. Latest consensus state: current_term: 1 leader_uuid: "41b1e853422246e9bb06f5c867029e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b1e853422246e9bb06f5c867029e12" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45361 } } }
I20260926 16:36:56.984407 10773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:56.984756 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41b1e853422246e9bb06f5c867029e12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b1e853422246e9bb06f5c867029e12" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45361 } } }
I20260926 16:36:56.984952 10767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:36:56.985491 10776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:36:56.987820 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12 [sys.catalog]: This master's current role is: LEADER
I20260926 16:36:56.990296 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:56.952877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.10.110.62:45361)
0926 16:36:56.953750 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:56.953770 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:56.954993 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 16:36:56.964219 (+ 9226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:56.964249 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:56.964810 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:56.965264 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:56.969111 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:56.970564 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:56.972315 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:56.973196 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:56.973402 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:56.978891 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:56.978899 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:56.978915 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:56.978955 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:56.982749 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:56.983819 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:56.983893 (+ 74us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:56.983901 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:56.984134 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:56.984766 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:56.984771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:56.984775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:56.985540 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:56.985870 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:56.988398 (+ 2528us) server_negotiation.cc:299] Negotiation successful
0926 16:36:56.988855 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":362,"threads_started":1}
I20260926 16:36:56.996464 10776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:36:57.004964 10776 catalog_manager.cc:1347] Generated new cluster ID: d681dec0fa7d4c839118b5cf32754022
I20260926 16:36:57.005092 10776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:36:57.020071 10776 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:36:57.021907 10776 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:36:57.029124 10776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41b1e853422246e9bb06f5c867029e12: Generated new TSK 0
I20260926 16:36:57.029960 10776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:36:57.071602 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:45361
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:57.385224 10793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:57.385540 10793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:57.385680 10793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:57.392974 10793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:57.393121 10793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:57.393278 10793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:36:57.403321 10793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:45361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:57.404677 10793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:57.405956 10793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:57.418171 10799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:57.418226 10798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:57.418361 10801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:57.418896 10793 server_base.cc:1034] running on GCE node
I20260926 16:36:57.419466 10793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:57.420451 10793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:57.421638 10793 hybrid_clock.cc:648] HybridClock initialized: now 1790440617421588 us; error 48 us; skew 500 ppm
I20260926 16:36:57.421927 10793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:57 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:57.424933 10793 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:36:57.426378 10793 webserver.cc:460] Webserver started at http://127.10.110.1:42119/ using document root <none> and password file <none>
I20260926 16:36:57.427031 10793 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:57.427151 10793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:57.427446 10793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:57.430387 10793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "75f06c707d7c4311a5bf20f230ab33d9"
format_stamp: "Formatted at 2026-09-26 16:36:57 on dist-test-slave-whjp"
server_key: "ea6a8456324178f146ea771565337bc5"
server_key_iv: "3c00024211769161f99c30d2a7e4c796"
server_key_version: "encryptionkey@0"
I20260926 16:36:57.431159 10793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "75f06c707d7c4311a5bf20f230ab33d9"
format_stamp: "Formatted at 2026-09-26 16:36:57 on dist-test-slave-whjp"
server_key: "ea6a8456324178f146ea771565337bc5"
server_key_iv: "3c00024211769161f99c30d2a7e4c796"
server_key_version: "encryptionkey@0"
I20260926 16:36:57.436129 10793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 16:36:57.439441 10808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:57.440758 10793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:36:57.440886 10793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "75f06c707d7c4311a5bf20f230ab33d9"
format_stamp: "Formatted at 2026-09-26 16:36:57 on dist-test-slave-whjp"
server_key: "ea6a8456324178f146ea771565337bc5"
server_key_iv: "3c00024211769161f99c30d2a7e4c796"
server_key_version: "encryptionkey@0"
I20260926 16:36:57.441041 10793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:57.457301 10793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:57.461483 10793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:57.461723 10793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:57.462661 10793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:36:57.463959 10793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:36:57.464035 10793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:57.464172 10793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:36:57.464243 10793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:57.493048 10793 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:43251
I20260926 16:36:57.493098 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:43251 every 8 connection(s)
I20260926 16:36:57.494283 10793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:36:57.497736 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 10793
I20260926 16:36:57.497941 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:36:57.498612 10680 external_mini_cluster.cc:1428] Setting key c040ae7c186b52db6cc05d3f4f1951ef
Sep 26 16:36:57 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:36:57.514003 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:57.496728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:38581 (local address 127.10.110.62:45361)
0926 16:36:57.497135 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:57.497140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:57.498262 (+ 1122us) server_negotiation.cc:407] Connection header received
0926 16:36:57.500290 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:57.500294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:57.500354 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:57.500447 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:57.501871 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:57.502837 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:57.504443 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:57.504859 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:57.504967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:57.508437 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:57.508442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:57.508445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:57.508480 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:57.510935 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:57.511668 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:57.511672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:57.511674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:57.511786 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:57.512252 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:57.512257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:57.512260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:57.512572 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:57.512693 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:57.513470 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 16:36:57.513659 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":203,"threads_started":1}
I20260926 16:36:57.514825 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:57.497302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:45361 (local address 127.10.110.1:38581)
0926 16:36:57.498055 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:57.498146 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:57.500026 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:57.500660 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:57.500667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:57.501156 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:57.501706 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:57.501723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:57.502988 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:57.502993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:57.504312 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:57.504324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:57.505069 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:57.505076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:57.505312 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:57.505983 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:57.506005 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:57.508202 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:57.511096 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:57.511106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:57.511126 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:57.511538 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:57.511910 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:57.511915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:57.511919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:57.512111 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:57.512674 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:57.512680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:57.513017 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 16:36:57.513301 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 16:36:57.513703 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":293,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:36:57.516359 10921 heartbeater.cc:347] Connected to a master server at 127.10.110.62:45361
I20260926 16:36:57.516711 10921 heartbeater.cc:464] Registering TS with master...
I20260926 16:36:57.517614 10921 heartbeater.cc:511] Master 127.10.110.62:45361 requested a full tablet report, sending...
I20260926 16:36:57.519757 10732 ts_manager.cc:194] Registered new tserver with Master: 75f06c707d7c4311a5bf20f230ab33d9 (127.10.110.1:43251)
I20260926 16:36:57.521193 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:38581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:36:57.535238 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:45361
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:57.848436 10927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:57.848703 10927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:57.848824 10927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:57.856082 10927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:57.856192 10927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:57.856352 10927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:36:57.866310 10927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:45361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:57.867594 10927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:57.869032 10927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:57.883751 10933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:57.883749 10932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:57.884517 10927 server_base.cc:1034] running on GCE node
W20260926 16:36:57.884819 10935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:57.885475 10927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:57.886471 10927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:57.887666 10927 hybrid_clock.cc:648] HybridClock initialized: now 1790440617887604 us; error 58 us; skew 500 ppm
I20260926 16:36:57.888001 10927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:57 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:57.891037 10927 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:36:57.892607 10927 webserver.cc:460] Webserver started at http://127.10.110.2:46733/ using document root <none> and password file <none>
I20260926 16:36:57.893301 10927 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:57.893415 10927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:57.893713 10927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:57.896684 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "2229c205d7a647a4abd6e8c7c1b6c667"
format_stamp: "Formatted at 2026-09-26 16:36:57 on dist-test-slave-whjp"
server_key: "deda53819c742a67ffb49792bc1f84b1"
server_key_iv: "fe9cfb78d81ae7fe701732b34b667fde"
server_key_version: "encryptionkey@0"
I20260926 16:36:57.897392 10927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "2229c205d7a647a4abd6e8c7c1b6c667"
format_stamp: "Formatted at 2026-09-26 16:36:57 on dist-test-slave-whjp"
server_key: "deda53819c742a67ffb49792bc1f84b1"
server_key_iv: "fe9cfb78d81ae7fe701732b34b667fde"
server_key_version: "encryptionkey@0"
I20260926 16:36:57.902578 10927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:57.906049 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:57.907534 10927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:36:57.907691 10927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "2229c205d7a647a4abd6e8c7c1b6c667"
format_stamp: "Formatted at 2026-09-26 16:36:57 on dist-test-slave-whjp"
server_key: "deda53819c742a67ffb49792bc1f84b1"
server_key_iv: "fe9cfb78d81ae7fe701732b34b667fde"
server_key_version: "encryptionkey@0"
I20260926 16:36:57.907886 10927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:57.946471 10927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:57.950729 10927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:57.950989 10927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:57.952018 10927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:36:57.953356 10927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:36:57.953441 10927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:57.953557 10927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:36:57.953608 10927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:57.992260 10927 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:43521
I20260926 16:36:57.992296 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:43521 every 8 connection(s)
I20260926 16:36:57.993637 10927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:36:58.000733 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 10927
I20260926 16:36:58.000934 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:36:58.001437 10680 external_mini_cluster.cc:1428] Setting key f4f079abb65e004dd59ebdb89635ae9b
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:36:58.017002 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:57.996471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:39641 (local address 127.10.110.62:45361)
0926 16:36:57.996637 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:57.996641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:57.998339 (+ 1698us) server_negotiation.cc:407] Connection header received
0926 16:36:58.000415 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.000420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.000528 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.000616 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.002544 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.003690 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.006135 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.006720 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.006811 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.010954 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.010961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.010967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.011005 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.013713 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.014458 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.014464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.014466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.014646 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.015147 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.015152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.015155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.015423 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.015517 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.016423 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.016657 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:36:58.018100 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:57.997267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:45361 (local address 127.10.110.2:39641)
0926 16:36:57.998172 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:57.998225 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:58.000134 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:58.000892 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:58.000904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:58.001460 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:58.002111 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:58.002143 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.003896 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.003903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:58.005963 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:58.005980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.007168 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.007174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:58.007451 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.008270 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:58.008302 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:58.010713 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:58.013885 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:58.013892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:58.013907 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:58.014319 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:58.014764 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:58.014769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:58.014772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:58.015016 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:58.015612 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:58.015620 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:58.016047 (+ 427us) client_negotiation.cc:769] Sending connection context
0926 16:36:58.016289 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 16:36:58.016868 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":328,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:36:58.019485 11055 heartbeater.cc:347] Connected to a master server at 127.10.110.62:45361
I20260926 16:36:58.019806 11055 heartbeater.cc:464] Registering TS with master...
I20260926 16:36:58.020581 11055 heartbeater.cc:511] Master 127.10.110.62:45361 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:36:58.022060 10732 ts_manager.cc:194] Registered new tserver with Master: 2229c205d7a647a4abd6e8c7c1b6c667 (127.10.110.2:43521)
I20260926 16:36:58.022846 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:39641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:36:58.042730 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:45361
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:36:58.353566 11060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:36:58.353849 11060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:36:58.353969 11060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:36:58.361305 11060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:36:58.361414 11060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:36:58.361562 11060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:36:58.372695 11060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:45361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:36:58.374372 11060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:36:58.375721 11060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:36:58.388418 11068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:58.388448 11065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:36:58.388479 11066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:36:58.389178 11060 server_base.cc:1034] running on GCE node
I20260926 16:36:58.389761 11060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:36:58.390736 11060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:36:58.391929 11060 hybrid_clock.cc:648] HybridClock initialized: now 1790440618391870 us; error 40 us; skew 500 ppm
I20260926 16:36:58.392261 11060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:36:58.395282 11060 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:36:58.396756 11060 webserver.cc:460] Webserver started at http://127.10.110.3:42389/ using document root <none> and password file <none>
I20260926 16:36:58.397415 11060 fs_manager.cc:359] Metadata directory not provided
I20260926 16:36:58.397526 11060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:36:58.397814 11060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:36:58.400785 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "a3d29abb76fe4b2e83481f167ebe48f5"
format_stamp: "Formatted at 2026-09-26 16:36:58 on dist-test-slave-whjp"
server_key: "d06c29f740591f7733f32125769f2ab7"
server_key_iv: "51150434348e276e99d7ad268f3d9510"
server_key_version: "encryptionkey@0"
I20260926 16:36:58.401485 11060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "a3d29abb76fe4b2e83481f167ebe48f5"
format_stamp: "Formatted at 2026-09-26 16:36:58 on dist-test-slave-whjp"
server_key: "d06c29f740591f7733f32125769f2ab7"
server_key_iv: "51150434348e276e99d7ad268f3d9510"
server_key_version: "encryptionkey@0"
I20260926 16:36:58.406533 11060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:36:58.409809 11075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:58.411160 11060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:36:58.411311 11060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "a3d29abb76fe4b2e83481f167ebe48f5"
format_stamp: "Formatted at 2026-09-26 16:36:58 on dist-test-slave-whjp"
server_key: "d06c29f740591f7733f32125769f2ab7"
server_key_iv: "51150434348e276e99d7ad268f3d9510"
server_key_version: "encryptionkey@0"
I20260926 16:36:58.411532 11060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:36:58.425557 11060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:36:58.429378 11060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:36:58.429630 11060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:36:58.430553 11060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:36:58.431871 11060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:36:58.431970 11060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:58.432088 11060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:36:58.432140 11060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:36:58.461122 11060 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:46139
I20260926 16:36:58.461167 11187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:46139 every 8 connection(s)
I20260926 16:36:58.462361 11060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:36:58.467962 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11060
I20260926 16:36:58.468158 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:36:58.468698 10680 external_mini_cluster.cc:1428] Setting key fa4603dd6a73355d19d90b0f5cb5009d
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:36:58.481593 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.464848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:52149 (local address 127.10.110.62:45361)
0926 16:36:58.465031 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:58.465037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:58.466379 (+ 1342us) server_negotiation.cc:407] Connection header received
0926 16:36:58.468310 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.468314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.468381 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.468472 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.470169 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.470953 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.473198 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.473587 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.473675 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.476714 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.476718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.476722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.476754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.479192 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.479809 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.479815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.479817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.479962 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.480293 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.480296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.480297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.480539 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.480639 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.481211 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.481360 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:36:58.482506 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.465373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:45361 (local address 127.10.110.3:52149)
0926 16:36:58.466231 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 16:36:58.466271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:36:58.468025 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:36:58.468739 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:36:58.468751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:36:58.469458 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:36:58.470035 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:58.470051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.471689 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.471695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:58.473065 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:36:58.473077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.473744 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.473748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:36:58.473927 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.474512 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:36:58.474533 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:36:58.476520 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:36:58.479315 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:58.479321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:58.479333 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:58.479721 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:58.480041 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:36:58.480044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:36:58.480046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:36:58.480210 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:36:58.480633 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:36:58.480638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:36:58.480923 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 16:36:58.481131 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 16:36:58.481500 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":335,"threads_started":1}
I20260926 16:36:58.483733 11188 heartbeater.cc:347] Connected to a master server at 127.10.110.62:45361
I20260926 16:36:58.484059 11188 heartbeater.cc:464] Registering TS with master...
I20260926 16:36:58.484740 11188 heartbeater.cc:511] Master 127.10.110.62:45361 requested a full tablet report, sending...
I20260926 16:36:58.485970 10732 ts_manager.cc:194] Registered new tserver with Master: a3d29abb76fe4b2e83481f167ebe48f5 (127.10.110.3:46139)
I20260926 16:36:58.486730 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:52149
I20260926 16:36:58.496610 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 16:36:58.524147 10921 heartbeater.cc:503] Master 127.10.110.62:45361 was elected leader, sending a full tablet report...
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:36:58.537811 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.523573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.10.110.1:43251)
0926 16:36:58.524194 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:58.524201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:58.524692 (+ 491us) server_negotiation.cc:407] Connection header received
0926 16:36:58.524828 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.524838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.525063 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.525211 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.526629 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.527598 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.528785 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.529328 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.529420 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.532816 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.532826 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.532840 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.532870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.535421 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.536098 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.536105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.536108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.536228 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.536561 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.536564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.536566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.536903 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.537121 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.537330 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.537523 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":307,"threads_started":1}
W20260926 16:36:58.538977 10836 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60432
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:36:58.578723 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.565354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.10.110.1:43251)
0926 16:36:58.565616 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:58.565623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:58.566077 (+ 454us) server_negotiation.cc:407] Connection header received
0926 16:36:58.566286 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.566290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.566357 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.566444 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.567757 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.568485 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.569689 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.570096 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.570204 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.573774 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.573779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.573782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.573816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.576319 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.576944 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.576956 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.576959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.577087 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.577418 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.577422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.577424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.577675 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.577897 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.578194 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.578414 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 16:36:58.580267 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 10793
I20260926 16:36:58.592355 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.581264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.10.110.1:43251)
0926 16:36:58.581426 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:58.581430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:58.581791 (+ 361us) server_negotiation.cc:407] Connection header received
0926 16:36:58.581900 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.581903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.581960 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.582054 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.583163 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.583886 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.585134 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.585579 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.585688 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.587805 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.587809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.587812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.587867 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.590129 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.590777 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.590781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.590783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.590908 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.591247 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.591249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.591251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.591473 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.591640 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.591944 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.592100 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:36:58.593106 10916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:58.673746 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 10793
I20260926 16:36:58.686905 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 10927
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:36:58.701737 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.687703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.10.110.2:43521)
0926 16:36:58.688226 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:58.688233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:58.688257 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:36:58.688315 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.688321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.688524 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.688678 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.689813 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.690816 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.692075 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.692630 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.692767 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.696602 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.696612 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.696627 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.696669 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.699288 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.699960 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.699967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.699970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.700111 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.700424 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.700426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.700428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.700732 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.700936 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.701192 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.701384 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260926 16:36:58.702898 11050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:58.782943 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 10927
I20260926 16:36:58.801401 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11060
Sep 26 16:36:58 dist-test-slave-whjp krb5kdc[10687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:36:58.816615 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:36:58.802353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.10.110.3:46139)
0926 16:36:58.802963 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0926 16:36:58.802970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:36:58.803003 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:36:58.803061 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:36:58.803067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:36:58.803280 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:36:58.803421 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:36:58.804667 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.805687 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.806920 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:36:58.807467 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:36:58.807578 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:36:58.811445 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:36:58.811456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:36:58.811471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:36:58.811506 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:36:58.813934 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.814550 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.814558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.814560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.814690 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:36:58.815049 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:36:58.815052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:36:58.815054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:36:58.815379 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:36:58.815670 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:36:58.815976 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 16:36:58.816226 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":314,"threads_started":1}
I20260926 16:36:58.817832 11183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:58.898777 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11060
I20260926 16:36:58.912132 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 10700
I20260926 16:36:58.913290 10762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:36:59.014330 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 10700
2026-09-26T16:36:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3652 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:36:59 dist-test-slave-whjp krb5kdc[11209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:36:59 dist-test-slave-whjp krb5kdc[11209](info): set up 2 sockets
Sep 26 16:36:59 dist-test-slave-whjp krb5kdc[11209](info): commencing operation
krb5kdc: starting...
W20260926 16:37:00.016458 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:00 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:00.179229 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33533
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:00.499565 11222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:00.499935 11222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:00.500082 11222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:00.507304 11222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:00.507413 11222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:00.507484 11222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:00.507562 11222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:00.507622 11222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:00.517584 11222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:00.518919 11222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:00.520288 11222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:00.531334 11227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:00.531276 11228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:00.531294 11230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:00.532729 11222 server_base.cc:1034] running on GCE node
I20260926 16:37:00.533460 11222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:00.534811 11222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:00.535996 11222 hybrid_clock.cc:648] HybridClock initialized: now 1790440620535945 us; error 39 us; skew 500 ppm
I20260926 16:37:00.536315 11222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:00 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440620, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:00.544231 11222 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:00.545863 11222 webserver.cc:460] Webserver started at http://127.10.110.62:33063/ using document root <none> and password file <none>
I20260926 16:37:00.546566 11222 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:00.546690 11222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:00.546990 11222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:00.550151 11222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "577a1fd59e984f6abced7859a765dcc2"
format_stamp: "Formatted at 2026-09-26 16:37:00 on dist-test-slave-whjp"
server_key: "e0fff73fc22e6c94b0dd5444c41d5b35"
server_key_iv: "bc21703d3792e11b30dccc0a7ea73a7d"
server_key_version: "encryptionkey@0"
I20260926 16:37:00.550951 11222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "577a1fd59e984f6abced7859a765dcc2"
format_stamp: "Formatted at 2026-09-26 16:37:00 on dist-test-slave-whjp"
server_key: "e0fff73fc22e6c94b0dd5444c41d5b35"
server_key_iv: "bc21703d3792e11b30dccc0a7ea73a7d"
server_key_version: "encryptionkey@0"
I20260926 16:37:00.556382 11222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:37:00.559998 11237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:00.561446 11222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:00.561640 11222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "577a1fd59e984f6abced7859a765dcc2"
format_stamp: "Formatted at 2026-09-26 16:37:00 on dist-test-slave-whjp"
server_key: "e0fff73fc22e6c94b0dd5444c41d5b35"
server_key_iv: "bc21703d3792e11b30dccc0a7ea73a7d"
server_key_version: "encryptionkey@0"
I20260926 16:37:00.561828 11222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:00.574988 11222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:00.579113 11222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:00.579402 11222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:00.596750 11222 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33533
I20260926 16:37:00.596786 11288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33533 every 8 connection(s)
I20260926 16:37:00.598111 11222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:00.601791 11289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:00.604925 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11222
I20260926 16:37:00.605150 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:00.605911 10680 external_mini_cluster.cc:1428] Setting key cad5dd15e80446be9af77e6eee37711f
I20260926 16:37:00.614279 11289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2: Bootstrap starting.
I20260926 16:37:00.618022 11289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:00.619313 11289 log.cc:824] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:37:00 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:00.622426 11289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2: No bootstrap required, opened a new log
I20260926 16:37:00.628517 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:00.607989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.10.110.62:33533)
0926 16:37:00.608714 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:00.608726 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:00.608768 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:37:00.610370 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:00.610393 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:00.610826 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:00.611149 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:00.612861 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:00.614148 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:00.615818 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:00.616553 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:00.616683 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:00.621346 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:00.621354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:00.621370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:00.621406 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:00.624373 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:00.625040 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:00.625047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:00.625054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:00.625234 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:00.625595 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:00.625599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:00.625601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:00.626129 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:00.626430 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:00.626933 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 16:37:00.627373 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":316,"threads_started":1}
I20260926 16:37:00.630134 11289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577a1fd59e984f6abced7859a765dcc2" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33533 } }
I20260926 16:37:00.630501 11289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:00.630596 11289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577a1fd59e984f6abced7859a765dcc2, State: Initialized, Role: FOLLOWER
I20260926 16:37:00.631265 11289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577a1fd59e984f6abced7859a765dcc2" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33533 } }
I20260926 16:37:00.631549 11289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:00.631708 11289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:00.631901 11289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:00.634075 11289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577a1fd59e984f6abced7859a765dcc2" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33533 } }
I20260926 16:37:00.634624 11289 leader_election.cc:302] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 577a1fd59e984f6abced7859a765dcc2; no voters:
I20260926 16:37:00.635267 11289 leader_election.cc:288] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:00.635360 11294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:00.636018 11294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [term 1 LEADER]: Becoming Leader. State: Replica: 577a1fd59e984f6abced7859a765dcc2, State: Running, Role: LEADER
I20260926 16:37:00.636636 11294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577a1fd59e984f6abced7859a765dcc2" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33533 } }
I20260926 16:37:00.637560 11289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:00.639935 11296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 577a1fd59e984f6abced7859a765dcc2. Latest consensus state: current_term: 1 leader_uuid: "577a1fd59e984f6abced7859a765dcc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577a1fd59e984f6abced7859a765dcc2" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33533 } } }
I20260926 16:37:00.640234 11296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:00.639695 11295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "577a1fd59e984f6abced7859a765dcc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "577a1fd59e984f6abced7859a765dcc2" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33533 } } }
I20260926 16:37:00.641248 11295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:00.645905 11300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:00.650645 11300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:00.661031 11300 catalog_manager.cc:1347] Generated new cluster ID: 9599baf69fe5451ab4a8450c6bd25d50
I20260926 16:37:00.661198 11300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:00.678905 11300 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:00.680791 11300 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:00.700287 11300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 577a1fd59e984f6abced7859a765dcc2: Generated new TSK 0
I20260926 16:37:00.701146 11300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:00.751410 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33533
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:01.063222 11315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:01.063486 11315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:01.063611 11315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:01.071066 11315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:01.071180 11315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:01.071329 11315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:01.080988 11315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:01.082243 11315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:01.083623 11315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:01.095922 11321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:01.095849 11320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:01.096063 11323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:01.096635 11315 server_base.cc:1034] running on GCE node
I20260926 16:37:01.097162 11315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:01.097998 11315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:01.099189 11315 hybrid_clock.cc:648] HybridClock initialized: now 1790440621099148 us; error 39 us; skew 500 ppm
I20260926 16:37:01.099452 11315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:01 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440621, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:01.102275 11315 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:01.103639 11315 webserver.cc:460] Webserver started at http://127.10.110.1:37467/ using document root <none> and password file <none>
I20260926 16:37:01.104312 11315 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:01.104422 11315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:01.104700 11315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:01.107728 11315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "68d276720b774bbfa82377be863efafa"
format_stamp: "Formatted at 2026-09-26 16:37:01 on dist-test-slave-whjp"
server_key: "adfa667dd834378623cb48a15c048ea4"
server_key_iv: "8e7b690cb90f302f9485318fe1847d54"
server_key_version: "encryptionkey@0"
I20260926 16:37:01.108546 11315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "68d276720b774bbfa82377be863efafa"
format_stamp: "Formatted at 2026-09-26 16:37:01 on dist-test-slave-whjp"
server_key: "adfa667dd834378623cb48a15c048ea4"
server_key_iv: "8e7b690cb90f302f9485318fe1847d54"
server_key_version: "encryptionkey@0"
I20260926 16:37:01.113416 11315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:37:01.116799 11330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:01.118076 11315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:01.118219 11315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "68d276720b774bbfa82377be863efafa"
format_stamp: "Formatted at 2026-09-26 16:37:01 on dist-test-slave-whjp"
server_key: "adfa667dd834378623cb48a15c048ea4"
server_key_iv: "8e7b690cb90f302f9485318fe1847d54"
server_key_version: "encryptionkey@0"
I20260926 16:37:01.118394 11315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:01.134488 11315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:01.138617 11315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:01.138882 11315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:01.139863 11315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:01.141153 11315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:01.141228 11315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:01.141345 11315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:01.141396 11315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:01.170547 11315 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:40395
I20260926 16:37:01.170600 11442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:40395 every 8 connection(s)
I20260926 16:37:01.171777 11315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:01.176677 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11315
I20260926 16:37:01.176857 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:01.177417 10680 external_mini_cluster.cc:1428] Setting key 87d04c57f21e1dac09e1628b762ea48e
Sep 26 16:37:01 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440621, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:01.192605 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:01.174349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48467 (local address 127.10.110.62:33533)
0926 16:37:01.174791 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:01.174796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:01.175672 (+ 876us) server_negotiation.cc:407] Connection header received
0926 16:37:01.177647 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:01.177653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:01.177740 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:01.177864 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:01.179588 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.180618 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.182155 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.182680 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.182790 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:01.186749 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:01.186754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:01.186757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:01.186791 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:01.189278 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:01.190201 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:01.190204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:01.190207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:01.190317 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:01.190809 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:01.190812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:01.190814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:01.191113 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:01.191222 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:01.192187 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 16:37:01.192348 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:37:01.194088 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:01.174809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33533 (local address 127.10.110.1:48467)
0926 16:37:01.175513 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:01.175553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:01.177353 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:01.178109 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:01.178122 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:01.178704 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:01.179414 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:01.179436 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.180766 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.180769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:01.182016 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:01.182028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.182882 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.182888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:01.183106 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:01.183774 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:01.183808 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:01.186521 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:01.189434 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:01.189444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:01.189462 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:01.190088 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:01.190430 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:01.190436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:01.190440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:01.190706 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:01.191240 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:01.191250 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:01.191732 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 16:37:01.192128 (+ 396us) client_negotiation.cc:240] Negotiation successful
0926 16:37:01.192695 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":261,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:37:01.195813 11443 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33533
I20260926 16:37:01.196213 11443 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:01.197165 11443 heartbeater.cc:511] Master 127.10.110.62:33533 requested a full tablet report, sending...
I20260926 16:37:01.199532 11254 ts_manager.cc:194] Registered new tserver with Master: 68d276720b774bbfa82377be863efafa (127.10.110.1:40395)
I20260926 16:37:01.201057 11254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:48467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:01.216215 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33533
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:01.521390 11449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:01.521670 11449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:01.521772 11449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:01.528963 11449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:01.529074 11449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:01.529206 11449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:01.539009 11449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:01.540347 11449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:01.541651 11449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:01.554153 11454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:01.554149 11455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:01.554359 11457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:01.555249 11449 server_base.cc:1034] running on GCE node
I20260926 16:37:01.555867 11449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:01.556787 11449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:01.557989 11449 hybrid_clock.cc:648] HybridClock initialized: now 1790440621557934 us; error 42 us; skew 500 ppm
I20260926 16:37:01.558300 11449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:01 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440621, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:01.561363 11449 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:01.562724 11449 webserver.cc:460] Webserver started at http://127.10.110.2:45991/ using document root <none> and password file <none>
I20260926 16:37:01.563387 11449 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:01.563494 11449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:01.563769 11449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:01.567036 11449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "192007f84f14459b9e2a44502e23c136"
format_stamp: "Formatted at 2026-09-26 16:37:01 on dist-test-slave-whjp"
server_key: "c86659da272e025ad13e158293947037"
server_key_iv: "2246cb6a6491c2f321655b7dae07b146"
server_key_version: "encryptionkey@0"
I20260926 16:37:01.567821 11449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "192007f84f14459b9e2a44502e23c136"
format_stamp: "Formatted at 2026-09-26 16:37:01 on dist-test-slave-whjp"
server_key: "c86659da272e025ad13e158293947037"
server_key_iv: "2246cb6a6491c2f321655b7dae07b146"
server_key_version: "encryptionkey@0"
I20260926 16:37:01.573354 11449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:37:01.576768 11464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:01.578078 11449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:01.578224 11449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "192007f84f14459b9e2a44502e23c136"
format_stamp: "Formatted at 2026-09-26 16:37:01 on dist-test-slave-whjp"
server_key: "c86659da272e025ad13e158293947037"
server_key_iv: "2246cb6a6491c2f321655b7dae07b146"
server_key_version: "encryptionkey@0"
I20260926 16:37:01.578408 11449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:01.592536 11449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:01.596848 11449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:01.597150 11449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:01.598261 11449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:01.599870 11449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:01.599961 11449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:37:01.600070 11449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:01.600126 11449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:01.630411 11449 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:33529
I20260926 16:37:01.630458 11576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:33529 every 8 connection(s)
I20260926 16:37:01.631675 11449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:01.641557 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11449
I20260926 16:37:01.641781 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:01.642372 10680 external_mini_cluster.cc:1428] Setting key e24c73f00d042870fb143fa8b9be5a1d
Sep 26 16:37:01 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440621, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:01.651542 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:01.634101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:55167 (local address 127.10.110.62:33533)
0926 16:37:01.634276 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:01.634281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:01.635412 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 16:37:01.637207 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:01.637212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:01.637282 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:01.637369 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:01.639002 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.639723 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.641314 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.641929 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.642053 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:01.645951 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:01.645956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:01.645959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:01.645993 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:01.648454 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:01.649318 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:01.649321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:01.649324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:01.649458 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:01.649872 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:01.649877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:01.649880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:01.650196 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:01.650354 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:01.651009 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 16:37:01.651232 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:37:01.652462 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:01.634578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33533 (local address 127.10.110.2:55167)
0926 16:37:01.635261 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:01.635299 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:01.636942 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:01.637595 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:01.637606 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:01.638205 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:01.638849 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:01.638868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.639907 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.639914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:01.641172 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:01.641187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:01.642142 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:01.642148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:01.642391 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:01.643123 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:01.643156 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:01.645725 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:01.648600 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:01.648610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:01.648630 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:01.649211 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:01.649564 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:01.649567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:01.649570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:01.649769 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:01.650302 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:01.650309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:01.650666 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 16:37:01.650883 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 16:37:01.651280 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":249,"threads_started":1}
I20260926 16:37:01.653831 11577 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33533
I20260926 16:37:01.654129 11577 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:01.654825 11577 heartbeater.cc:511] Master 127.10.110.62:33533 requested a full tablet report, sending...
I20260926 16:37:01.656207 11254 ts_manager.cc:194] Registered new tserver with Master: 192007f84f14459b9e2a44502e23c136 (127.10.110.2:33529)
I20260926 16:37:01.656957 11254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:55167
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:01.679179 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33533
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:01.993106 11582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:01.993393 11582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:01.993512 11582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:02.000861 11582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:02.000998 11582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:02.001165 11582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:02.011373 11582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33533
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:02.012718 11582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:02.014038 11582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:02.026865 11587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:02.026839 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:02.026803 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:02.027483 11582 server_base.cc:1034] running on GCE node
I20260926 16:37:02.028142 11582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:02.029083 11582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:02.030282 11582 hybrid_clock.cc:648] HybridClock initialized: now 1790440622030229 us; error 47 us; skew 500 ppm
I20260926 16:37:02.030596 11582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:02.037793 11582 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:02.039400 11582 webserver.cc:460] Webserver started at http://127.10.110.3:35659/ using document root <none> and password file <none>
I20260926 16:37:02.040164 11582 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:02.040279 11582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:02.040565 11582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:02.043555 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "d7d3ec2b42be4346aaaa4a360049225e"
format_stamp: "Formatted at 2026-09-26 16:37:02 on dist-test-slave-whjp"
server_key: "3de983793e2fa5173ff2c66810e9dff1"
server_key_iv: "6bf7b54a6a0c216c8c8453d742287c3c"
server_key_version: "encryptionkey@0"
I20260926 16:37:02.044344 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "d7d3ec2b42be4346aaaa4a360049225e"
format_stamp: "Formatted at 2026-09-26 16:37:02 on dist-test-slave-whjp"
server_key: "3de983793e2fa5173ff2c66810e9dff1"
server_key_iv: "6bf7b54a6a0c216c8c8453d742287c3c"
server_key_version: "encryptionkey@0"
I20260926 16:37:02.049325 11582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 16:37:02.052690 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:02.053946 11582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:02.054091 11582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "d7d3ec2b42be4346aaaa4a360049225e"
format_stamp: "Formatted at 2026-09-26 16:37:02 on dist-test-slave-whjp"
server_key: "3de983793e2fa5173ff2c66810e9dff1"
server_key_iv: "6bf7b54a6a0c216c8c8453d742287c3c"
server_key_version: "encryptionkey@0"
I20260926 16:37:02.054272 11582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:02.068971 11582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:02.072808 11582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:02.073067 11582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:02.074033 11582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:02.075330 11582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:02.075405 11582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:02.075519 11582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:02.075574 11582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:02.104111 11582 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:36491
I20260926 16:37:02.104153 11709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:36491 every 8 connection(s)
I20260926 16:37:02.105394 11582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:02.109412 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11582
I20260926 16:37:02.109601 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:02.110016 10680 external_mini_cluster.cc:1428] Setting key 17c3a95314058f3d15d8ec423ac3f5db
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:02.124176 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.107876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:37267 (local address 127.10.110.62:33533)
0926 16:37:02.108039 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.108044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.109232 (+ 1188us) server_negotiation.cc:407] Connection header received
0926 16:37:02.111061 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.111066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.111152 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.111235 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.112870 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.113614 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.115196 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.115593 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.115707 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.118922 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.118927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.118931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.118962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.121214 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.121935 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.121939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.121941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.122057 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.122466 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.122469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.122471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.122756 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.122883 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.123575 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.123809 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:37:02.125202 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.108380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33533 (local address 127.10.110.3:37267)
0926 16:37:02.109069 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:02.109119 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:02.110801 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:02.111507 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:02.111519 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:02.112120 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:02.112706 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.112725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.113775 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.113778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.115049 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.115062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.115818 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.115825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.116086 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.116683 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:02.116705 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:02.118715 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:02.121343 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:02.121353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:02.121372 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:02.121814 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:02.122160 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:02.122165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:02.122169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:02.122353 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:02.122875 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:02.122883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:02.123229 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 16:37:02.123458 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 16:37:02.124022 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":272,"threads_started":1}
I20260926 16:37:02.126412 11710 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33533
I20260926 16:37:02.126704 11710 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:02.127393 11710 heartbeater.cc:511] Master 127.10.110.62:33533 requested a full tablet report, sending...
I20260926 16:37:02.128716 11254 ts_manager.cc:194] Registered new tserver with Master: d7d3ec2b42be4346aaaa4a360049225e (127.10.110.3:36491)
I20260926 16:37:02.129523 11254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:37267
I20260926 16:37:02.137493 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:37:02.155570 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.144015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.10.110.62:33533)
0926 16:37:02.144218 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.144226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.144667 (+ 441us) server_negotiation.cc:407] Connection header received
0926 16:37:02.144808 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.144812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.144897 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.145028 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.146436 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.147184 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.148455 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.148900 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.149057 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.151170 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.151174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.151178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.151211 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.153432 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.153995 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.153998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.154000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.154130 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.154391 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.154394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.154395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.154680 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.154851 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.155122 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.155281 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:37:02.161720 11254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:02.168434 11254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:02.189683 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.180532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.10.110.2:33529)
0926 16:37:02.181036 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.181044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.181560 (+ 516us) server_negotiation.cc:407] Connection header received
0926 16:37:02.181677 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.181684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.181931 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.182110 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:02.183016 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.184847 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.187240 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.188505 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.188623 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.188813 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.189181 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.189346 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260926 16:37:02.189728 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.180434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33529 (local address 127.0.0.1:45614)
0926 16:37:02.181136 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:02.181172 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:02.181338 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:02.182366 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:02.182373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:02.182405 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:02.182816 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.182828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.185001 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.185006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.187097 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.187109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.188705 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.188712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.188938 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.188988 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 16:37:02.189124 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 16:37:02.189304 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":20,"thread_start_us":614,"threads_started":1}
I20260926 16:37:02.191359 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.181721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:36491 (local address 127.0.0.1:40994)
0926 16:37:02.182244 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:02.182260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:02.182362 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:02.184197 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:02.184201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:02.184225 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:02.184567 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.184578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.186731 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.186736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.188918 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.188927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.190557 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.190564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.190728 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.190761 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:02.190846 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 16:37:02.190975 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":54,"thread_start_us":240,"threads_started":1}
I20260926 16:37:02.191911 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.182779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.10.110.1:40395)
0926 16:37:02.184117 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.184124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.184149 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:37:02.184235 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.184242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.184447 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.185142 (+ 695us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:02.185310 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.186242 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.189028 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.190408 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.190533 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.190731 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.191447 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.191594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":866,"threads_started":1}
I20260926 16:37:02.192206 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.182244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:40395 (local address 127.0.0.1:39378)
0926 16:37:02.182850 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:02.182868 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:02.182968 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:02.184642 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:02.184647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:02.184667 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:02.184995 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.185005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.186368 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.186374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.188868 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.188886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.190861 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.190867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.191033 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.191064 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:37:02.191162 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:37:02.191270 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":76,"thread_start_us":685,"threads_started":1}
I20260926 16:37:02.192837 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.182072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.10.110.3:36491)
0926 16:37:02.183606 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.183614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.183648 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:37:02.183742 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.183749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.183992 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.184767 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:02.185026 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.186597 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.189061 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.190385 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.190500 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.190704 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.190899 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.191042 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":1202,"threads_started":1}
I20260926 16:37:02.194042 11512 tablet_service.cc:1467] Processing CreateTablet for tablet 292dd042f57541419fa162e6387c05b1 (DEFAULT_TABLE table=test-table [id=0439918ae05d46a38860bcbbd9be0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:02.195518 11512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292dd042f57541419fa162e6387c05b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:02.195431 11378 tablet_service.cc:1467] Processing CreateTablet for tablet 292dd042f57541419fa162e6387c05b1 (DEFAULT_TABLE table=test-table [id=0439918ae05d46a38860bcbbd9be0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:02.196655 11378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292dd042f57541419fa162e6387c05b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:02.196164 11645 tablet_service.cc:1467] Processing CreateTablet for tablet 292dd042f57541419fa162e6387c05b1 (DEFAULT_TABLE table=test-table [id=0439918ae05d46a38860bcbbd9be0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:02.197428 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292dd042f57541419fa162e6387c05b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:02.203708 11443 heartbeater.cc:503] Master 127.10.110.62:33533 was elected leader, sending a full tablet report...
I20260926 16:37:02.207901 11728 tablet_bootstrap.cc:492] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: Bootstrap starting.
I20260926 16:37:02.211295 11728 tablet_bootstrap.cc:654] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:02.212486 11728 log.cc:824] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:02.212565 11729 tablet_bootstrap.cc:492] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: Bootstrap starting.
I20260926 16:37:02.216241 11729 tablet_bootstrap.cc:654] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:02.216876 11730 tablet_bootstrap.cc:492] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: Bootstrap starting.
I20260926 16:37:02.217482 11729 log.cc:824] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:02.218438 11728 tablet_bootstrap.cc:492] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: No bootstrap required, opened a new log
I20260926 16:37:02.218783 11728 ts_tablet_manager.cc:1397] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260926 16:37:02.220129 11730 tablet_bootstrap.cc:654] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:02.220485 11729 tablet_bootstrap.cc:492] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: No bootstrap required, opened a new log
I20260926 16:37:02.221117 11730 log.cc:824] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:02.221529 11729 ts_tablet_manager.cc:1397] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260926 16:37:02.223694 11730 tablet_bootstrap.cc:492] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: No bootstrap required, opened a new log
I20260926 16:37:02.223944 11730 ts_tablet_manager.cc:1397] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260926 16:37:02.228875 11728 raft_consensus.cc:348] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.229328 11728 raft_consensus.cc:374] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:02.229432 11728 raft_consensus.cc:729] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d276720b774bbfa82377be863efafa, State: Initialized, Role: FOLLOWER
I20260926 16:37:02.230065 11728 consensus_queue.cc:260] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.231120 11730 raft_consensus.cc:348] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.231562 11730 raft_consensus.cc:374] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:02.231701 11730 raft_consensus.cc:729] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d3ec2b42be4346aaaa4a360049225e, State: Initialized, Role: FOLLOWER
I20260926 16:37:02.231493 11729 raft_consensus.cc:348] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.231998 11729 raft_consensus.cc:374] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:02.232121 11729 raft_consensus.cc:729] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192007f84f14459b9e2a44502e23c136, State: Initialized, Role: FOLLOWER
I20260926 16:37:02.232177 11728 ts_tablet_manager.cc:1428] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 16:37:02.232486 11730 consensus_queue.cc:260] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.232709 11729 consensus_queue.cc:260] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.233517 11710 heartbeater.cc:503] Master 127.10.110.62:33533 was elected leader, sending a full tablet report...
I20260926 16:37:02.234046 11730 ts_tablet_manager.cc:1428] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260926 16:37:02.234771 11577 heartbeater.cc:503] Master 127.10.110.62:33533 was elected leader, sending a full tablet report...
I20260926 16:37:02.235365 11729 ts_tablet_manager.cc:1428] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
W20260926 16:37:02.356899 11711 tablet.cc:2369] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:02.361135 11734 raft_consensus.cc:482] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:02.361414 11734 raft_consensus.cc:504] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.363026 11734 leader_election.cc:288] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 192007f84f14459b9e2a44502e23c136 (127.10.110.2:33529), d7d3ec2b42be4346aaaa4a360049225e (127.10.110.3:36491)
I20260926 16:37:02.370416 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.363432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:51875 (local address 127.10.110.2:33529)
0926 16:37:02.363605 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.363611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.364156 (+ 545us) server_negotiation.cc:407] Connection header received
0926 16:37:02.364396 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.364403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.364500 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.364600 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:02.365183 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.365992 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.368126 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.369225 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.369377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.369454 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.369991 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.370143 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:37:02.370661 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.364320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:41607 (local address 127.10.110.3:36491)
0926 16:37:02.364542 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.364548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.364869 (+ 321us) server_negotiation.cc:407] Connection header received
0926 16:37:02.365071 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.365077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.365149 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.365238 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:02.365741 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.366729 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.368646 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.369651 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.369810 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.369875 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.370295 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.370413 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:37:02.370740 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.364171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:36491 (local address 127.10.110.1:41607)
0926 16:37:02.364752 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:02.364770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:02.364893 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:02.365264 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:02.365268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:02.365291 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:02.365599 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.365607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.366868 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.366872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.368474 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.368487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.370011 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.370018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.370146 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.370168 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:37:02.370247 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:37:02.370371 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":3,"thread_start_us":278,"threads_started":1}
I20260926 16:37:02.371488 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.363373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33529 (local address 127.10.110.1:51875)
0926 16:37:02.364031 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:02.364048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:02.364212 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:02.364639 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:02.364644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:02.364672 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:02.365025 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.365039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.366119 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.366125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.367956 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:02.367968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.369597 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.369603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:02.369801 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.369837 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:37:02.370965 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0926 16:37:02.371124 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":27,"thread_start_us":342,"threads_started":1}
I20260926 16:37:02.372339 11665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292dd042f57541419fa162e6387c05b1" candidate_uuid: "68d276720b774bbfa82377be863efafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d3ec2b42be4346aaaa4a360049225e" is_pre_election: true
I20260926 16:37:02.372750 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292dd042f57541419fa162e6387c05b1" candidate_uuid: "68d276720b774bbfa82377be863efafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "192007f84f14459b9e2a44502e23c136" is_pre_election: true
I20260926 16:37:02.373260 11532 raft_consensus.cc:2454] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68d276720b774bbfa82377be863efafa in term 0.
I20260926 16:37:02.373282 11665 raft_consensus.cc:2454] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68d276720b774bbfa82377be863efafa in term 0.
I20260926 16:37:02.374048 11334 leader_election.cc:302] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192007f84f14459b9e2a44502e23c136, 68d276720b774bbfa82377be863efafa; no voters:
I20260926 16:37:02.374476 11734 raft_consensus.cc:2781] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:02.374627 11734 raft_consensus.cc:482] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:02.374724 11734 raft_consensus.cc:3037] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:02.377873 11734 raft_consensus.cc:504] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.378595 11734 leader_election.cc:288] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [CANDIDATE]: Term 1 election: Requested vote from peers 192007f84f14459b9e2a44502e23c136 (127.10.110.2:33529), d7d3ec2b42be4346aaaa4a360049225e (127.10.110.3:36491)
I20260926 16:37:02.379088 11532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292dd042f57541419fa162e6387c05b1" candidate_uuid: "68d276720b774bbfa82377be863efafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "192007f84f14459b9e2a44502e23c136"
I20260926 16:37:02.379410 11532 raft_consensus.cc:3037] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:02.379963 11665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292dd042f57541419fa162e6387c05b1" candidate_uuid: "68d276720b774bbfa82377be863efafa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d3ec2b42be4346aaaa4a360049225e"
I20260926 16:37:02.380152 11665 raft_consensus.cc:3037] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:02.381963 11532 raft_consensus.cc:2454] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68d276720b774bbfa82377be863efafa in term 1.
I20260926 16:37:02.382275 11665 raft_consensus.cc:2454] T 292dd042f57541419fa162e6387c05b1 P d7d3ec2b42be4346aaaa4a360049225e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68d276720b774bbfa82377be863efafa in term 1.
I20260926 16:37:02.382529 11334 leader_election.cc:302] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192007f84f14459b9e2a44502e23c136, 68d276720b774bbfa82377be863efafa; no voters:
I20260926 16:37:02.382823 11734 raft_consensus.cc:2781] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:02.383314 11734 raft_consensus.cc:686] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [term 1 LEADER]: Becoming Leader. State: Replica: 68d276720b774bbfa82377be863efafa, State: Running, Role: LEADER
W20260926 16:37:02.383692 11578 tablet.cc:2369] T 292dd042f57541419fa162e6387c05b1 P 192007f84f14459b9e2a44502e23c136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:02.384151 11734 consensus_queue.cc:237] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } }
I20260926 16:37:02.394999 11254 catalog_manager.cc:5495] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa reported cstate change: term changed from 0 to 1, leader changed from <none> to 68d276720b774bbfa82377be863efafa (127.10.110.1). New cstate: current_term: 1 leader_uuid: "68d276720b774bbfa82377be863efafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "192007f84f14459b9e2a44502e23c136" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d3ec2b42be4346aaaa4a360049225e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 36491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68d276720b774bbfa82377be863efafa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40395 } health_report { overall_health: HEALTHY } } }
W20260926 16:37:02.424364 11444 tablet.cc:2369] T 292dd042f57541419fa162e6387c05b1 P 68d276720b774bbfa82377be863efafa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:02.430688 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.417809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39392 (local address 127.10.110.1:40395)
0926 16:37:02.418032 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.418038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.418319 (+ 281us) server_negotiation.cc:407] Connection header received
0926 16:37:02.418446 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.418450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.418515 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.418616 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.419927 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.420622 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.421874 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.422242 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.422344 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.426010 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.426022 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.426037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.426068 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.428528 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.429089 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.429096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.429098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.429240 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.429498 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.429501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.429503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.429830 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.430032 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.430229 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.430385 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:02.470949 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.458126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.10.110.1:40395)
0926 16:37:02.458296 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.458303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.458652 (+ 349us) server_negotiation.cc:407] Connection header received
0926 16:37:02.458830 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.458835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.458900 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.458981 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.460278 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.460996 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.462209 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.462627 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.462734 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.466315 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.466320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.466342 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.466375 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.468555 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.469116 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.469122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.469126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.469245 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.469610 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.469615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.469619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.469889 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.470169 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.470416 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.470634 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:02.511505 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.498738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.10.110.1:40395)
0926 16:37:02.498980 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.498987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.499171 (+ 184us) server_negotiation.cc:407] Connection header received
0926 16:37:02.499445 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.499450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.499549 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.499649 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.500974 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.501793 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.503001 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.503421 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.503527 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.507030 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.507034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.507037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.507071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.509469 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.510025 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.510029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.510031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.510157 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.510422 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.510425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.510427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.510671 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.510842 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.511017 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.511189 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260926 16:37:02.513162 11358 tablet_service.cc:2908] Rejecting scan request for tablet 292dd042f57541419fa162e6387c05b1: Uninitialized: clean time has not yet been initialized
I20260926 16:37:02.516278 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11315
I20260926 16:37:02.527490 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.517112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.10.110.1:40395)
0926 16:37:02.517273 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.517277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.517584 (+ 307us) server_negotiation.cc:407] Connection header received
0926 16:37:02.517734 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.517738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.517807 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.517879 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.518898 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.519559 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.520662 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.521033 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.521119 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.523143 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.523148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.523151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.523178 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.525505 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.526076 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.526079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.526081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.526192 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.526445 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.526448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.526450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.526680 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.526840 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.527063 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.527239 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:37:02.528118 11438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:02.617430 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11315
I20260926 16:37:02.630759 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11449
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:02.647995 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.631685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.10.110.2:33529)
0926 16:37:02.631859 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.631864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.631887 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:37:02.631945 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.631948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.632009 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.632099 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.633531 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.634317 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.636151 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.636551 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.636650 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.642047 (+ 5397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.642062 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.642077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.642130 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.645384 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.646115 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.646123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.646125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.646246 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.646683 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.646685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.646687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.647034 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.647282 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.647511 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.647671 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:37:02.648597 11572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:02.736421 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11449
I20260926 16:37:02.749274 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11582
Sep 26 16:37:02 dist-test-slave-whjp krb5kdc[11209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:02.762981 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:02.750030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.10.110.3:36491)
0926 16:37:02.750216 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:02.750222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:02.750242 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:37:02.750423 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:02.750427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:02.750491 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:02.750574 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:02.751750 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.752506 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.753757 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:02.754145 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:02.754250 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:02.758307 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:02.758322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:02.758339 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:02.758373 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:02.760811 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.761389 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.761396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.761398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.761522 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:02.761783 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:02.761786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:02.761788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:02.762119 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:02.762319 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:02.762586 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:37:02.762735 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:37:02.763573 11705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:02.848665 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11582
I20260926 16:37:02.860948 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11222
I20260926 16:37:02.862005 11284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:02.967411 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11222
2026-09-26T16:37:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3956 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:03 dist-test-slave-whjp krb5kdc[11758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:03 dist-test-slave-whjp krb5kdc[11758](info): set up 2 sockets
Sep 26 16:37:03 dist-test-slave-whjp krb5kdc[11758](info): commencing operation
krb5kdc: starting...
W20260926 16:37:03.952500 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:04 dist-test-slave-whjp krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:04.122861 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40149
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:40149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:04.429864 11771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:04.430161 11771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:04.430305 11771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:04.437503 11771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:04.437608 11771 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:37:04.437673 11771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:04.437759 11771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:04.437822 11771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:04.437901 11771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:04.447888 11771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:40149
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:04.449151 11771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:04.450460 11771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:04.463078 11779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:04.463171 11776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:04.463178 11777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:04.464102 11771 server_base.cc:1034] running on GCE node
I20260926 16:37:04.464829 11771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:04.466166 11771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:04.467371 11771 hybrid_clock.cc:648] HybridClock initialized: now 1790440624467319 us; error 31 us; skew 500 ppm
I20260926 16:37:04.467680 11771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:04 dist-test-slave-whjp krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440624, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:04.470783 11771 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:04.472239 11771 webserver.cc:460] Webserver started at http://127.10.110.62:38445/ using document root <none> and password file <none>
I20260926 16:37:04.472927 11771 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:04.473022 11771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:04.473315 11771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:04.476155 11771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "f43107f7089c47458b532ce5468e45ce"
format_stamp: "Formatted at 2026-09-26 16:37:04 on dist-test-slave-whjp"
server_key: "b1577d78241b9359408c1bd50a7f77b4"
server_key_iv: "62a3b7abbda163f9028d3be4c5647d0b"
server_key_version: "encryptionkey@0"
I20260926 16:37:04.476877 11771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "f43107f7089c47458b532ce5468e45ce"
format_stamp: "Formatted at 2026-09-26 16:37:04 on dist-test-slave-whjp"
server_key: "b1577d78241b9359408c1bd50a7f77b4"
server_key_iv: "62a3b7abbda163f9028d3be4c5647d0b"
server_key_version: "encryptionkey@0"
I20260926 16:37:04.481990 11771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:37:04.485457 11786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:04.486829 11771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:37:04.486989 11771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "f43107f7089c47458b532ce5468e45ce"
format_stamp: "Formatted at 2026-09-26 16:37:04 on dist-test-slave-whjp"
server_key: "b1577d78241b9359408c1bd50a7f77b4"
server_key_iv: "62a3b7abbda163f9028d3be4c5647d0b"
server_key_version: "encryptionkey@0"
I20260926 16:37:04.487192 11771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:04.539651 11771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:04.543568 11771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:04.543817 11771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:04.563215 11771 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:40149
I20260926 16:37:04.563230 11847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:40149 every 8 connection(s)
I20260926 16:37:04.564528 11771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:04.568507 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:04.568833 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11771
I20260926 16:37:04.569054 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:04.569893 10680 external_mini_cluster.cc:1428] Setting key 9b7d57520e31b9736aa631ff20555d9e
I20260926 16:37:04.582566 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: Bootstrap starting.
Sep 26 16:37:04 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:04.586268 11848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:04.587313 11848 log.cc:824] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:04.590554 11848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: No bootstrap required, opened a new log
I20260926 16:37:04.592885 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:04.572124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.10.110.62:40149)
0926 16:37:04.572927 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:04.572939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:04.572986 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:37:04.574792 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:04.574815 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:04.575252 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:04.575587 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:04.577184 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:04.578453 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:04.580156 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:04.580859 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:04.580988 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:04.585623 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:04.585631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:04.585647 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:04.585686 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:04.588874 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:04.589654 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:04.589660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:04.589666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:04.589846 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:04.590231 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:04.590236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:04.590240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:04.590798 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:04.591042 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:04.591461 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 16:37:04.591819 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":352,"threads_started":1}
I20260926 16:37:04.598438 11848 raft_consensus.cc:348] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43107f7089c47458b532ce5468e45ce" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40149 } }
I20260926 16:37:04.598784 11848 raft_consensus.cc:374] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:04.598877 11848 raft_consensus.cc:729] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43107f7089c47458b532ce5468e45ce, State: Initialized, Role: FOLLOWER
I20260926 16:37:04.599419 11848 consensus_queue.cc:260] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43107f7089c47458b532ce5468e45ce" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40149 } }
I20260926 16:37:04.599638 11848 raft_consensus.cc:388] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:04.599756 11848 raft_consensus.cc:482] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:04.599916 11848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:04.601992 11848 raft_consensus.cc:504] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43107f7089c47458b532ce5468e45ce" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40149 } }
I20260926 16:37:04.602461 11848 leader_election.cc:302] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f43107f7089c47458b532ce5468e45ce; no voters:
I20260926 16:37:04.603076 11848 leader_election.cc:288] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:04.603193 11853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:04.603812 11853 raft_consensus.cc:686] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [term 1 LEADER]: Becoming Leader. State: Replica: f43107f7089c47458b532ce5468e45ce, State: Running, Role: LEADER
I20260926 16:37:04.604343 11853 consensus_queue.cc:237] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43107f7089c47458b532ce5468e45ce" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40149 } }
I20260926 16:37:04.605042 11848 sys_catalog.cc:564] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:04.607851 11854 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f43107f7089c47458b532ce5468e45ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43107f7089c47458b532ce5468e45ce" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40149 } } }
I20260926 16:37:04.608341 11854 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:04.608238 11855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f43107f7089c47458b532ce5468e45ce. Latest consensus state: current_term: 1 leader_uuid: "f43107f7089c47458b532ce5468e45ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43107f7089c47458b532ce5468e45ce" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40149 } } }
I20260926 16:37:04.608487 11855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:04.609439 11862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:04.614959 11862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:04.624199 11862 catalog_manager.cc:1347] Generated new cluster ID: 791ba49b215b4746a72930eaafc4f1e0
I20260926 16:37:04.624318 11862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:04.637643 11862 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:04.639499 11862 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:04.662696 11862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: Generated new TSK 0
I20260926 16:37:04.663786 11862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:04.708428 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40149
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:05.016234 11874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:05.016525 11874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:05.016675 11874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:05.023802 11874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:05.023948 11874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:05.024076 11874 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:05.024189 11874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:05.033886 11874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:05.035141 11874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:05.036419 11874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:05.048355 11879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:05.048513 11882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:05.048583 11880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:05.049508 11874 server_base.cc:1034] running on GCE node
I20260926 16:37:05.050066 11874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:05.050889 11874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:05.052100 11874 hybrid_clock.cc:648] HybridClock initialized: now 1790440625052031 us; error 39 us; skew 500 ppm
I20260926 16:37:05.052372 11874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:05 dist-test-slave-whjp krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440625, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:05.055284 11874 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:05.056640 11874 webserver.cc:460] Webserver started at http://127.10.110.1:37789/ using document root <none> and password file <none>
I20260926 16:37:05.057407 11874 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:05.057497 11874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:05.057782 11874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:05.060700 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "9945c7937ad84c7e8b453da503f1d517"
format_stamp: "Formatted at 2026-09-26 16:37:05 on dist-test-slave-whjp"
server_key: "2e398ef86afc19aa125f583b32c5a878"
server_key_iv: "b54e060b73406d43eefdd51b871d84e8"
server_key_version: "encryptionkey@0"
I20260926 16:37:05.061445 11874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "9945c7937ad84c7e8b453da503f1d517"
format_stamp: "Formatted at 2026-09-26 16:37:05 on dist-test-slave-whjp"
server_key: "2e398ef86afc19aa125f583b32c5a878"
server_key_iv: "b54e060b73406d43eefdd51b871d84e8"
server_key_version: "encryptionkey@0"
I20260926 16:37:05.066562 11874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:37:05.070048 11889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:05.071362 11874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:05.071504 11874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "9945c7937ad84c7e8b453da503f1d517"
format_stamp: "Formatted at 2026-09-26 16:37:05 on dist-test-slave-whjp"
server_key: "2e398ef86afc19aa125f583b32c5a878"
server_key_iv: "b54e060b73406d43eefdd51b871d84e8"
server_key_version: "encryptionkey@0"
I20260926 16:37:05.071691 11874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:05.095250 11874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:05.099170 11874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:05.099406 11874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:05.101981 11874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:05.102098 11874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:05.102236 11874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:05.102301 11874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:05 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440625, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:05.125449 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.102932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:58935 (local address 127.10.110.62:40149)
0926 16:37:05.103365 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:05.103370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:05.104568 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 16:37:05.106676 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:05.106682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:05.106771 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:05.106874 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:05.109641 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.110672 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.113160 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.113737 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.113868 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.118943 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:05.118950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:05.118955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:05.119008 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:05.122223 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.122950 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.122956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.122960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.123097 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.123536 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.123541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.123544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.123806 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:05.123973 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:05.124885 (+ 912us) server_negotiation.cc:299] Negotiation successful
0926 16:37:05.125100 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260926 16:37:05.127008 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.103429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.10.110.1:58935)
0926 16:37:05.104404 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:05.104448 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:05.106386 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:05.107409 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:05.107426 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:05.108400 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:05.109399 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.109428 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.110850 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.110855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.112990 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.113004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.114027 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.114037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.114319 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.115193 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:05.115243 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:05.118662 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:05.122363 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.122373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.122392 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.122820 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.123203 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.123212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.123216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.123416 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.123943 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:05.123955 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:05.124464 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 16:37:05.124783 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 16:37:05.125391 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":402,"threads_started":1}
I20260926 16:37:05.147656 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.134160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:43769 (local address 127.10.110.62:40149)
0926 16:37:05.134355 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:05.134362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:05.134907 (+ 545us) server_negotiation.cc:407] Connection header received
0926 16:37:05.135084 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:05.135088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:05.135150 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:05.135231 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:05.136375 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.137507 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.139149 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.139722 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.139875 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.142208 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:05.142213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:05.142216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:05.142254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:05.144893 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.145580 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.145583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.145586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.145719 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.146131 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.146134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.146136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.146411 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:05.146643 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:05.147215 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 16:37:05.147397 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:37:05.148047 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.133813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.10.110.1:43769)
0926 16:37:05.134799 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:05.134818 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:05.134968 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:05.135266 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:05.135270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:05.135731 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:05.136154 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.136166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.137681 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.137687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.139003 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.139019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.139969 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.139976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.140186 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.140855 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:05.140878 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:05.142002 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:05.145068 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.145073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.145076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.145444 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.145836 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.145843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.145847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.146030 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.146638 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:05.146646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:05.146917 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 16:37:05.147102 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 16:37:05.147504 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":692,"threads_started":1}
I20260926 16:37:05.152766 11874 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:33389
I20260926 16:37:05.152786 12009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:33389 every 8 connection(s)
I20260926 16:37:05.153928 11874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:05.155473 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 11874
I20260926 16:37:05.155704 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:05.155871 12010 heartbeater.cc:347] Connected to a master server at 127.10.110.62:40149
I20260926 16:37:05.156085 12010 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:05.156337 10680 external_mini_cluster.cc:1428] Setting key 0413a4d240d633803875721118ef8252
I20260926 16:37:05.156713 12010 heartbeater.cc:511] Master 127.10.110.62:40149 requested a full tablet report, sending...
I20260926 16:37:05.158524 11802 ts_manager.cc:194] Registered new tserver with Master: 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389)
I20260926 16:37:05.160002 11802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:58935
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:05.191172 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40149
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:05.497211 12014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:05.497480 12014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:05.497603 12014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:05.504703 12014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:05.504809 12014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:05.504922 12014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:05.505021 12014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:05.514665 12014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:05.515913 12014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:05.517230 12014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:05.529374 12022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:05.529399 12019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:05.529448 12020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:05.530241 12014 server_base.cc:1034] running on GCE node
I20260926 16:37:05.530840 12014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:05.531778 12014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:05.533016 12014 hybrid_clock.cc:648] HybridClock initialized: now 1790440625532951 us; error 38 us; skew 500 ppm
I20260926 16:37:05.533311 12014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:05 dist-test-slave-whjp krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440625, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:05.536245 12014 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:05.537577 12014 webserver.cc:460] Webserver started at http://127.10.110.2:45351/ using document root <none> and password file <none>
I20260926 16:37:05.538251 12014 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:05.538345 12014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:05.538647 12014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:05.541455 12014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2"
format_stamp: "Formatted at 2026-09-26 16:37:05 on dist-test-slave-whjp"
server_key: "77039b4b578fc3846d6cc4beda5c0349"
server_key_iv: "b7a079d0d3d79d5af9f440761bf1f085"
server_key_version: "encryptionkey@0"
I20260926 16:37:05.542197 12014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2"
format_stamp: "Formatted at 2026-09-26 16:37:05 on dist-test-slave-whjp"
server_key: "77039b4b578fc3846d6cc4beda5c0349"
server_key_iv: "b7a079d0d3d79d5af9f440761bf1f085"
server_key_version: "encryptionkey@0"
I20260926 16:37:05.546998 12014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 16:37:05.550280 12029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:05.551569 12014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:05.551712 12014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2"
format_stamp: "Formatted at 2026-09-26 16:37:05 on dist-test-slave-whjp"
server_key: "77039b4b578fc3846d6cc4beda5c0349"
server_key_iv: "b7a079d0d3d79d5af9f440761bf1f085"
server_key_version: "encryptionkey@0"
I20260926 16:37:05.551962 12014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:05.570837 12014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:05.575150 12014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:05.575385 12014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:05.578147 12014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:05.578251 12014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:05.578390 12014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:05.578456 12014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:05 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440625, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:05.598551 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.578729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:43343 (local address 127.10.110.62:40149)
0926 16:37:05.578927 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:05.578935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:05.580573 (+ 1638us) server_negotiation.cc:407] Connection header received
0926 16:37:05.582746 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:05.582752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:05.582839 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:05.582933 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:05.585333 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.586069 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.587784 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.588316 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.588449 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.592696 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:05.592701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:05.592704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:05.592737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:05.595027 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.595762 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.595767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.595771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.595958 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.596453 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.596459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.596462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.596858 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:05.597015 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:05.598006 (+ 991us) server_negotiation.cc:299] Negotiation successful
0926 16:37:05.598225 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:37:05.600018 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.579485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.10.110.2:43343)
0926 16:37:05.580410 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:05.580454 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:05.582456 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:05.583337 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:05.583346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:05.584314 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:05.585150 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.585180 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.586229 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.586234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.587572 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.587584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.588644 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.588657 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.588953 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.589745 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:05.589783 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:05.592477 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:05.595160 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.595169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.595185 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.595632 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.596097 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.596104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.596107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.596317 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.596997 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:05.597005 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:05.597504 (+ 499us) client_negotiation.cc:769] Sending connection context
0926 16:37:05.597846 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 16:37:05.598392 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":397,"threads_started":1}
I20260926 16:37:05.619184 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.605538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.10.110.2:60231)
0926 16:37:05.606170 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:05.606192 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:05.606358 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:05.606653 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:05.606658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:05.607231 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:05.607611 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.607623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.608746 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.608752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.610320 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:05.610338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.611036 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.611042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:05.611223 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.611821 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:05.611870 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:05.612800 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:05.616310 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.616315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.616319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.616957 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.617360 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:05.617366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:05.617370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:05.617543 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:05.618207 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:05.618210 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:05.618365 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 16:37:05.618477 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 16:37:05.618760 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":325,"threads_started":1}
I20260926 16:37:05.619208 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:05.605516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:60231 (local address 127.10.110.62:40149)
0926 16:37:05.605705 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:05.605710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:05.606274 (+ 564us) server_negotiation.cc:407] Connection header received
0926 16:37:05.606463 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:05.606466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:05.606526 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:05.606605 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:05.607734 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.608623 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.610464 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:05.610871 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:05.610972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:05.613030 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:05.613037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:05.613042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:05.613081 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:05.616188 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.617089 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.617095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.617099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.617249 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:05.617661 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:05.617666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:05.617669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:05.618099 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:05.618251 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:05.618638 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 16:37:05.618886 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:37:05.627614 12014 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:37659
I20260926 16:37:05.627647 12148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:37659 every 8 connection(s)
I20260926 16:37:05.628793 12014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:05.630743 12149 heartbeater.cc:347] Connected to a master server at 127.10.110.62:40149
I20260926 16:37:05.630954 12149 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:05.631578 12149 heartbeater.cc:511] Master 127.10.110.62:40149 requested a full tablet report, sending...
I20260926 16:37:05.632846 11802 ts_manager.cc:194] Registered new tserver with Master: 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659)
I20260926 16:37:05.633656 11802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:43343
I20260926 16:37:05.636247 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 12014
I20260926 16:37:05.636416 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:05.636816 10680 external_mini_cluster.cc:1428] Setting key 5d29b1617da5e9ae4746ee94f0762963
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:05.672413 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40149
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:05.981588 12153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:05.981884 12153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:05.982026 12153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:05.989250 12153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:05.989358 12153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:05.989463 12153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:05.989564 12153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:06.000443 12153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:40149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:06.001784 12153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:06.003140 12153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:06.015475 12158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:06.015507 12159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:06.015628 12161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:06.016290 12153 server_base.cc:1034] running on GCE node
I20260926 16:37:06.016867 12153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:06.017781 12153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:06.018977 12153 hybrid_clock.cc:648] HybridClock initialized: now 1790440626018906 us; error 54 us; skew 500 ppm
I20260926 16:37:06.019279 12153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:06 dist-test-slave-whjp krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:06.022188 12153 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:06.023564 12153 webserver.cc:460] Webserver started at http://127.10.110.3:45161/ using document root <none> and password file <none>
I20260926 16:37:06.024247 12153 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:06.024330 12153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:06.024577 12153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:06.027318 12153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b"
format_stamp: "Formatted at 2026-09-26 16:37:06 on dist-test-slave-whjp"
server_key: "95e9748d78da17eab80582686c785e69"
server_key_iv: "2371768ca78ea97dd16eb69780c17ad9"
server_key_version: "encryptionkey@0"
I20260926 16:37:06.028043 12153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b"
format_stamp: "Formatted at 2026-09-26 16:37:06 on dist-test-slave-whjp"
server_key: "95e9748d78da17eab80582686c785e69"
server_key_iv: "2371768ca78ea97dd16eb69780c17ad9"
server_key_version: "encryptionkey@0"
I20260926 16:37:06.032985 12153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:37:06.036192 12168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:06.037535 12153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:37:06.037653 12153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b"
format_stamp: "Formatted at 2026-09-26 16:37:06 on dist-test-slave-whjp"
server_key: "95e9748d78da17eab80582686c785e69"
server_key_iv: "2371768ca78ea97dd16eb69780c17ad9"
server_key_version: "encryptionkey@0"
I20260926 16:37:06.037819 12153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:06.055632 12153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:06.059433 12153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:06.059659 12153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:06.062564 12153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:06.062674 12153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:06.062876 12153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:06.062958 12153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:06 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:06.085538 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.063666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.10.110.3:54857)
0926 16:37:06.064520 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.064559 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.066440 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.067211 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.067219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.068021 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:06.068873 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.068909 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.070457 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.070462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.072505 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.072517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.073437 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.073444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.073764 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.074597 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:06.074638 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:06.077434 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:06.081606 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:06.081614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:06.081628 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:06.082236 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:06.082611 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:06.082616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:06.082620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:06.082819 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:06.083337 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:06.083345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:06.083709 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.084001 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.084469 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":382,"threads_started":1}
I20260926 16:37:06.086426 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.062945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:54857 (local address 127.10.110.62:40149)
0926 16:37:06.063106 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.063111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.064684 (+ 1573us) server_negotiation.cc:407] Connection header received
0926 16:37:06.066747 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.066753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.066836 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.066922 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:06.069097 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.070282 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.072673 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.073205 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.073326 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.077738 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:06.077746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:06.077751 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:06.077824 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:06.081433 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.082370 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.082376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.082378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.082507 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.082937 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.082941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.082943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.083219 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:06.083350 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.085999 (+ 2649us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.086181 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:37:06.108462 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.095182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:43703 (local address 127.10.110.62:40149)
0926 16:37:06.095351 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.095357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.096119 (+ 762us) server_negotiation.cc:407] Connection header received
0926 16:37:06.096276 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.096281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.096346 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.096425 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:06.097319 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.098057 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.099312 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.099689 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.099792 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.102278 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:06.102285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:06.102289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:06.102329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:06.105546 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.106263 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.106269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.106274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.106441 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.106909 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.106915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.106919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.107288 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:06.107455 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.108034 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.108214 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:37:06.108819 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.095254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.10.110.3:43703)
0926 16:37:06.096009 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.096029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.096161 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.096461 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.096465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.096900 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:06.097178 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.097188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.098180 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.098185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.099182 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.099194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.099877 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.099884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.100046 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.100695 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:06.100721 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:06.102061 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:06.105683 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:06.105688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:06.105691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:06.106148 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:06.106587 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:06.106593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:06.106597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:06.106776 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:06.107416 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:06.107422 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:06.107708 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.107931 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.108333 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":365,"threads_started":1}
I20260926 16:37:06.116516 12153 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:35405
I20260926 16:37:06.116539 12287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:35405 every 8 connection(s)
I20260926 16:37:06.117772 12153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:06.119657 12288 heartbeater.cc:347] Connected to a master server at 127.10.110.62:40149
I20260926 16:37:06.119874 12288 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:06.120468 12288 heartbeater.cc:511] Master 127.10.110.62:40149 requested a full tablet report, sending...
I20260926 16:37:06.121699 11802 ts_manager.cc:194] Registered new tserver with Master: 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405)
I20260926 16:37:06.122455 11802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:54857
I20260926 16:37:06.128118 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 12153
I20260926 16:37:06.128268 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:06.128640 10680 external_mini_cluster.cc:1428] Setting key bfc35ea752f03dc0922fa84246527443
I20260926 16:37:06.131469 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:37:06 dist-test-slave-whjp krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 16:37:06.162798 12010 heartbeater.cc:503] Master 127.10.110.62:40149 was elected leader, sending a full tablet report...
Sep 26 16:37:06 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:06.174700 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.160434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.10.110.62:40149)
0926 16:37:06.160654 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.160661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.160894 (+ 233us) server_negotiation.cc:407] Connection header received
0926 16:37:06.161069 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.161073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.161149 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.161233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:06.162522 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.163490 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.165043 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.165467 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.165574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.169395 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:06.169400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:06.169403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:06.169439 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:06.171994 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.172741 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.172746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.172748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.172890 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.173215 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.173219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.173221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.173556 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:06.173739 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.174118 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.174301 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:37:06.179160 11802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:06.185959 11802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:06.225065 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.198176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37659 (local address 127.0.0.1:42638)
0926 16:37:06.216588 (+ 18412us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.216608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.216945 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.218017 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.218025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.218057 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.218462 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.218474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.220187 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.220192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.222319 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.222330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.224112 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.224119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.224304 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.224355 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.224474 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.224671 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18289,"mutex_wait_us":86,"thread_start_us":19531,"threads_started":1}
I20260926 16:37:06.225270 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.205273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.10.110.1:33389)
0926 16:37:06.216121 (+ 10848us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.216132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.218248 (+ 2116us) server_negotiation.cc:407] Connection header received
0926 16:37:06.218354 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.218361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.218618 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.218792 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.219327 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.220660 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.222917 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.224312 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.224435 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.224625 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.224863 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.225021 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10687,"thread_start_us":10518,"threads_started":1}
I20260926 16:37:06.225842 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.198676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.10.110.2:37659)
0926 16:37:06.212458 (+ 13782us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.212469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.216801 (+ 4332us) server_negotiation.cc:407] Connection header received
0926 16:37:06.217153 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.217161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.217532 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.217753 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.218633 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.220027 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.222464 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.223607 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.223735 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.223973 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.225505 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.225646 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13576,"thread_start_us":13795,"threads_started":1}
I20260926 16:37:06.225368 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.210688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33389 (local address 127.0.0.1:36038)
0926 16:37:06.217996 (+ 7308us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.218012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.218116 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.218792 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.218796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.218819 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.219187 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.219195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.220759 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.220764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.222763 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.222777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.224502 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.224508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.224678 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.224708 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.224803 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.224930 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7175,"mutex_wait_us":47,"thread_start_us":5611,"threads_started":1}
I20260926 16:37:06.225875 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.197572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:35405 (local address 127.0.0.1:48926)
0926 16:37:06.204550 (+ 6978us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.204609 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.204806 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.218954 (+ 14148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.218959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.218989 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.219388 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.219399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.221562 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.221568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.223528 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.223540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.225286 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.225292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.225466 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.225496 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.225546 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.225627 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":3,"thread_start_us":518,"threads_started":1}
I20260926 16:37:06.226802 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.197819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.10.110.3:35405)
0926 16:37:06.218392 (+ 20573us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.218398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.218424 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:37:06.218504 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.218509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.218718 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.219592 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.219876 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.221421 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.223696 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.225081 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.225988 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.226294 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.226368 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.226512 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20435,"thread_start_us":20284,"threads_started":1}
I20260926 16:37:06.229722 12079 tablet_service.cc:1467] Processing CreateTablet for tablet fd1786a890374eafbe2b78fa08321907 (DEFAULT_TABLE table=test-table [id=24cdaa7cfdfc47dfb7982040e84dd45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:06.229722 11940 tablet_service.cc:1467] Processing CreateTablet for tablet fd1786a890374eafbe2b78fa08321907 (DEFAULT_TABLE table=test-table [id=24cdaa7cfdfc47dfb7982040e84dd45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:06.231083 11940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1786a890374eafbe2b78fa08321907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.231098 12079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1786a890374eafbe2b78fa08321907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.230775 12218 tablet_service.cc:1467] Processing CreateTablet for tablet fd1786a890374eafbe2b78fa08321907 (DEFAULT_TABLE table=test-table [id=24cdaa7cfdfc47dfb7982040e84dd45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:06.231789 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1786a890374eafbe2b78fa08321907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.243089 12307 tablet_bootstrap.cc:492] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Bootstrap starting.
I20260926 16:37:06.247176 12307 tablet_bootstrap.cc:654] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.248231 12307 log.cc:824] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:06.250512 12309 tablet_bootstrap.cc:492] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Bootstrap starting.
I20260926 16:37:06.251181 12307 tablet_bootstrap.cc:492] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: No bootstrap required, opened a new log
I20260926 16:37:06.251430 12307 ts_tablet_manager.cc:1397] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:37:06.253389 12309 tablet_bootstrap.cc:654] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.253618 12308 tablet_bootstrap.cc:492] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Bootstrap starting.
I20260926 16:37:06.254479 12309 log.cc:824] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:06.257202 12308 tablet_bootstrap.cc:654] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.258415 12308 log.cc:824] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:06.260571 12307 raft_consensus.cc:348] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.260916 12307 raft_consensus.cc:374] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.261030 12307 raft_consensus.cc:729] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c34f761d8246fd9a3f7eb67a1c5f8b, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.261510 12308 tablet_bootstrap.cc:492] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: No bootstrap required, opened a new log
I20260926 16:37:06.261514 12309 tablet_bootstrap.cc:492] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: No bootstrap required, opened a new log
I20260926 16:37:06.261610 12307 consensus_queue.cc:260] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.261794 12308 ts_tablet_manager.cc:1397] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260926 16:37:06.261794 12309 ts_tablet_manager.cc:1397] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 16:37:06.262624 12288 heartbeater.cc:503] Master 127.10.110.62:40149 was elected leader, sending a full tablet report...
I20260926 16:37:06.264362 12307 ts_tablet_manager.cc:1428] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 16:37:06.270344 12309 raft_consensus.cc:348] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.270694 12309 raft_consensus.cc:374] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.270792 12309 raft_consensus.cc:729] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9945c7937ad84c7e8b453da503f1d517, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.271327 12309 consensus_queue.cc:260] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.271433 12308 raft_consensus.cc:348] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.271901 12308 raft_consensus.cc:374] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.272003 12308 raft_consensus.cc:729] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1dcd836a6c4cb0bac13c21bce07ad2, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.272666 12308 consensus_queue.cc:260] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.272950 12309 ts_tablet_manager.cc:1428] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260926 16:37:06.273635 12149 heartbeater.cc:503] Master 127.10.110.62:40149 was elected leader, sending a full tablet report...
I20260926 16:37:06.274209 12308 ts_tablet_manager.cc:1428] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
W20260926 16:37:06.369403 12289 tablet.cc:2369] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:06.375032 12314 raft_consensus.cc:482] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:06.375365 12314 raft_consensus.cc:504] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.377118 12314 leader_election.cc:288] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405), 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659)
W20260926 16:37:06.380518 12150 tablet.cc:2369] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:06.383912 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.377991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:36231 (local address 127.10.110.3:35405)
0926 16:37:06.378221 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.378227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.378249 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:37:06.378412 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.378418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.378480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.378560 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.379149 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.380073 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.382052 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.382941 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.383058 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.383111 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.383488 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.383628 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:37:06.383991 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.377552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:35405 (local address 127.10.110.1:36231)
0926 16:37:06.378147 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.378161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.378268 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.378594 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.378598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.378623 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.378998 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.379011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.380189 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.380194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.381909 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.381920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.383166 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.383172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.383336 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.383369 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.383468 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.383607 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":49,"thread_start_us":296,"threads_started":1}
I20260926 16:37:06.384263 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.377619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:50385 (local address 127.10.110.2:37659)
0926 16:37:06.377825 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.377831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.378774 (+ 943us) server_negotiation.cc:407] Connection header received
0926 16:37:06.378904 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.378909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.378980 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.379069 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.379605 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.380466 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.382490 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.383372 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.383562 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.383635 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.383936 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.384061 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:37:06.384320 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.377562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37659 (local address 127.10.110.1:50385)
0926 16:37:06.378679 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.378693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.378805 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.379095 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.379099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.379146 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.379473 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.379484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.380614 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.380619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.382320 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.382334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.383587 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.383593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.383750 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.383782 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.383911 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.384034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":22,"thread_start_us":820,"threads_started":1}
I20260926 16:37:06.385219 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1786a890374eafbe2b78fa08321907" candidate_uuid: "9945c7937ad84c7e8b453da503f1d517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" is_pre_election: true
I20260926 16:37:06.385268 12238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1786a890374eafbe2b78fa08321907" candidate_uuid: "9945c7937ad84c7e8b453da503f1d517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" is_pre_election: true
I20260926 16:37:06.385669 12104 raft_consensus.cc:2454] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9945c7937ad84c7e8b453da503f1d517 in term 0.
I20260926 16:37:06.385682 12238 raft_consensus.cc:2454] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9945c7937ad84c7e8b453da503f1d517 in term 0.
I20260926 16:37:06.386257 11893 leader_election.cc:302] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1dcd836a6c4cb0bac13c21bce07ad2, 9945c7937ad84c7e8b453da503f1d517; no voters:
I20260926 16:37:06.386610 12314 raft_consensus.cc:2781] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:06.386783 12314 raft_consensus.cc:482] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:06.386894 12314 raft_consensus.cc:3037] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.389288 12314 raft_consensus.cc:504] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.390004 12314 leader_election.cc:288] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [CANDIDATE]: Term 1 election: Requested vote from peers 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405), 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659)
I20260926 16:37:06.390357 12238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1786a890374eafbe2b78fa08321907" candidate_uuid: "9945c7937ad84c7e8b453da503f1d517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b"
I20260926 16:37:06.390496 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1786a890374eafbe2b78fa08321907" candidate_uuid: "9945c7937ad84c7e8b453da503f1d517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2"
I20260926 16:37:06.390606 12238 raft_consensus.cc:3037] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.390699 12104 raft_consensus.cc:3037] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.393841 12238 raft_consensus.cc:2454] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9945c7937ad84c7e8b453da503f1d517 in term 1.
I20260926 16:37:06.393841 12104 raft_consensus.cc:2454] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9945c7937ad84c7e8b453da503f1d517 in term 1.
I20260926 16:37:06.394258 11893 leader_election.cc:302] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17c34f761d8246fd9a3f7eb67a1c5f8b, 7b1dcd836a6c4cb0bac13c21bce07ad2, 9945c7937ad84c7e8b453da503f1d517; no voters:
I20260926 16:37:06.394580 12314 raft_consensus.cc:2781] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:06.395036 12314 raft_consensus.cc:686] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 1 LEADER]: Becoming Leader. State: Replica: 9945c7937ad84c7e8b453da503f1d517, State: Running, Role: LEADER
I20260926 16:37:06.395480 12314 consensus_queue.cc:237] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } }
I20260926 16:37:06.401177 11802 catalog_manager.cc:5495] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "9945c7937ad84c7e8b453da503f1d517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } health_report { overall_health: UNKNOWN } } }
W20260926 16:37:06.406252 12011 tablet.cc:2369] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:06.470678 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.465559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.10.110.1:33389)
0926 16:37:06.465723 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.465728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.465747 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:37:06.465980 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.465983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.466073 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.466188 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:06.466463 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.467174 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.468478 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.468867 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.468955 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.469422 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.470049 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.470175 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.470285 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.470358 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:37:06.478595 12238 raft_consensus.cc:1261] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Refusing update from remote peer 9945c7937ad84c7e8b453da503f1d517: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:06.478595 12104 raft_consensus.cc:1261] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Refusing update from remote peer 9945c7937ad84c7e8b453da503f1d517: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:06.479609 12318 consensus_queue.cc:1035] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:06.479945 12314 consensus_queue.cc:1035] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:06.510916 11803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:06.511242 11803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:06.515477 11803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: Sending DeleteTablet for 3 replicas of tablet fd1786a890374eafbe2b78fa08321907
I20260926 16:37:06.516798 12218 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1786a890374eafbe2b78fa08321907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:06 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:48926
I20260926 16:37:06.517182 11940 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1786a890374eafbe2b78fa08321907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:06 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:36038
I20260926 16:37:06.517519 12079 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1786a890374eafbe2b78fa08321907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:06 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42638
I20260926 16:37:06.517709 12331 tablet_replica.cc:330] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: stopping tablet replica
I20260926 16:37:06.518415 12331 raft_consensus.cc:2229] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:06.518580 11803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:37:06.518951 12331 raft_consensus.cc:2258] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 16:37:06.519460 11803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:06.519944 12332 tablet_replica.cc:330] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: stopping tablet replica
I20260926 16:37:06.520412 12332 raft_consensus.cc:2229] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:06.521070 12332 raft_consensus.cc:2258] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:06.521350 12331 ts_tablet_manager.cc:1905] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:06.522245 12333 tablet_replica.cc:330] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: stopping tablet replica
I20260926 16:37:06.526683 11940 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4c4ec30430491b9341f2b0e36034e9 (DEFAULT_TABLE table=test-table [id=bd3d87192ca14f0c89a9a83bf3c35628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:06.527139 11940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4c4ec30430491b9341f2b0e36034e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.527137 12218 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4c4ec30430491b9341f2b0e36034e9 (DEFAULT_TABLE table=test-table [id=bd3d87192ca14f0c89a9a83bf3c35628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:06.527230 12078 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4c4ec30430491b9341f2b0e36034e9 (DEFAULT_TABLE table=test-table [id=bd3d87192ca14f0c89a9a83bf3c35628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:06.527551 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4c4ec30430491b9341f2b0e36034e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.527707 12078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4c4ec30430491b9341f2b0e36034e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.532333 12331 ts_tablet_manager.cc:1918] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:06.532572 12331 log.cc:1196] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/fd1786a890374eafbe2b78fa08321907
I20260926 16:37:06.532689 12308 tablet_bootstrap.cc:492] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Bootstrap starting.
I20260926 16:37:06.533247 12309 tablet_bootstrap.cc:492] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Bootstrap starting.
I20260926 16:37:06.533260 12331 ts_tablet_manager.cc:1939] T fd1786a890374eafbe2b78fa08321907 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Deleting consensus metadata
I20260926 16:37:06.534286 12308 tablet_bootstrap.cc:654] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.534853 12309 tablet_bootstrap.cc:654] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.535241 12307 tablet_bootstrap.cc:492] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Bootstrap starting.
I20260926 16:37:06.536397 11788 catalog_manager.cc:4841] TS 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405): tablet fd1786a890374eafbe2b78fa08321907 (table test-table [id=24cdaa7cfdfc47dfb7982040e84dd45a]) successfully deleted
I20260926 16:37:06.537124 12309 tablet_bootstrap.cc:492] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: No bootstrap required, opened a new log
I20260926 16:37:06.537264 12309 ts_tablet_manager.cc:1397] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260926 16:37:06.537432 12307 tablet_bootstrap.cc:654] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.538076 12309 raft_consensus.cc:348] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.538322 12309 raft_consensus.cc:374] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.538424 12309 raft_consensus.cc:729] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9945c7937ad84c7e8b453da503f1d517, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.538679 12309 consensus_queue.cc:260] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.539252 12309 ts_tablet_manager.cc:1428] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 16:37:06.539727 12333 raft_consensus.cc:2229] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:06.540490 12333 raft_consensus.cc:2258] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:06.540738 12308 tablet_bootstrap.cc:492] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: No bootstrap required, opened a new log
I20260926 16:37:06.540868 12308 ts_tablet_manager.cc:1397] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20260926 16:37:06.541534 12307 tablet_bootstrap.cc:492] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: No bootstrap required, opened a new log
I20260926 16:37:06.541683 12307 ts_tablet_manager.cc:1397] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260926 16:37:06.541618 12308 raft_consensus.cc:348] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.541839 12308 raft_consensus.cc:374] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.541934 12308 raft_consensus.cc:729] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1dcd836a6c4cb0bac13c21bce07ad2, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.542186 12308 consensus_queue.cc:260] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.542397 12307 raft_consensus.cc:348] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.542606 12307 raft_consensus.cc:374] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.542688 12307 raft_consensus.cc:729] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c34f761d8246fd9a3f7eb67a1c5f8b, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.542829 12308 ts_tablet_manager.cc:1428] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 16:37:06.543246 12307 consensus_queue.cc:260] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.544358 12307 ts_tablet_manager.cc:1428] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 16:37:06.547468 12333 ts_tablet_manager.cc:1905] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:06.549340 12332 ts_tablet_manager.cc:1905] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:06.552237 12313 raft_consensus.cc:482] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:06.552409 12313 raft_consensus.cc:504] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.553891 12313 leader_election.cc:288] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389), 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659)
I20260926 16:37:06.554610 12333 ts_tablet_manager.cc:1918] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:06.554785 12333 log.cc:1196] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/fd1786a890374eafbe2b78fa08321907
I20260926 16:37:06.555318 12333 ts_tablet_manager.cc:1939] T fd1786a890374eafbe2b78fa08321907 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Deleting consensus metadata
I20260926 16:37:06.557237 11790 catalog_manager.cc:4841] TS 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659): tablet fd1786a890374eafbe2b78fa08321907 (table test-table [id=24cdaa7cfdfc47dfb7982040e84dd45a]) successfully deleted
I20260926 16:37:06.561522 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.554087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33389 (local address 127.10.110.3:43199)
0926 16:37:06.555167 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.555186 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.555303 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.555883 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.555888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.555934 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.556119 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.556131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.557511 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.557517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.558916 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.558929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.560671 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.560677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.560878 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.560911 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.560998 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.561136 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957}
I20260926 16:37:06.561549 12332 ts_tablet_manager.cc:1918] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:06.561710 12332 log.cc:1196] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/fd1786a890374eafbe2b78fa08321907
I20260926 16:37:06.562271 12332 ts_tablet_manager.cc:1939] T fd1786a890374eafbe2b78fa08321907 P 9945c7937ad84c7e8b453da503f1d517: Deleting consensus metadata
I20260926 16:37:06.564041 11788 catalog_manager.cc:4841] TS 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389): tablet fd1786a890374eafbe2b78fa08321907 (table test-table [id=24cdaa7cfdfc47dfb7982040e84dd45a]) successfully deleted
I20260926 16:37:06.564980 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.554392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:43199 (local address 127.10.110.1:33389)
0926 16:37:06.554665 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.554672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.555409 (+ 737us) server_negotiation.cc:407] Connection header received
0926 16:37:06.555481 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.555486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.555585 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.555702 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.556245 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.557394 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.559065 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.560530 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.560671 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.560740 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.562482 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.562596 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260926 16:37:06.565943 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4c4ec30430491b9341f2b0e36034e9" candidate_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9945c7937ad84c7e8b453da503f1d517" is_pre_election: true
I20260926 16:37:06.566403 11960 raft_consensus.cc:2454] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c34f761d8246fd9a3f7eb67a1c5f8b in term 0.
I20260926 16:37:06.567173 12170 leader_election.cc:302] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c34f761d8246fd9a3f7eb67a1c5f8b, 9945c7937ad84c7e8b453da503f1d517; no voters:
I20260926 16:37:06.568238 12313 raft_consensus.cc:2781] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:06.568367 12313 raft_consensus.cc:482] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:06.568461 12313 raft_consensus.cc:3037] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.570120 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.554801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:39759 (local address 127.10.110.2:37659)
0926 16:37:06.554996 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.555001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.561839 (+ 6838us) server_negotiation.cc:407] Connection header received
0926 16:37:06.562213 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.562219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.562284 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.562332 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.563322 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.564454 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.566703 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.567908 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.568043 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.568116 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.569670 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.569833 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:37:06.570906 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.554291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37659 (local address 127.10.110.3:39759)
0926 16:37:06.561753 (+ 7462us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.561772 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.562054 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.562970 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.562975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.563001 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.563157 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.563166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.564637 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.564642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.566505 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.566519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.569208 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.569219 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.569454 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.569498 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.570440 (+ 942us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.570594 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7374,"thread_start_us":10434,"threads_started":1}
I20260926 16:37:06.571659 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4c4ec30430491b9341f2b0e36034e9" candidate_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" is_pre_election: true
I20260926 16:37:06.571622 12313 raft_consensus.cc:504] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.571965 12104 raft_consensus.cc:2454] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c34f761d8246fd9a3f7eb67a1c5f8b in term 0.
I20260926 16:37:06.572685 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4c4ec30430491b9341f2b0e36034e9" candidate_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9945c7937ad84c7e8b453da503f1d517"
I20260926 16:37:06.572913 11960 raft_consensus.cc:3037] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.573249 12313 leader_election.cc:288] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [CANDIDATE]: Term 1 election: Requested vote from peers 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389), 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659)
I20260926 16:37:06.573830 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4c4ec30430491b9341f2b0e36034e9" candidate_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2"
I20260926 16:37:06.574028 12104 raft_consensus.cc:3037] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.576205 11960 raft_consensus.cc:2454] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c34f761d8246fd9a3f7eb67a1c5f8b in term 1.
I20260926 16:37:06.576684 12170 leader_election.cc:302] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c34f761d8246fd9a3f7eb67a1c5f8b, 9945c7937ad84c7e8b453da503f1d517; no voters:
I20260926 16:37:06.576990 12313 raft_consensus.cc:2781] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:06.578471 12104 raft_consensus.cc:2454] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c34f761d8246fd9a3f7eb67a1c5f8b in term 1.
I20260926 16:37:06.579304 12313 raft_consensus.cc:686] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 LEADER]: Becoming Leader. State: Replica: 17c34f761d8246fd9a3f7eb67a1c5f8b, State: Running, Role: LEADER
I20260926 16:37:06.579984 12313 consensus_queue.cc:237] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.583635 11803 catalog_manager.cc:5495] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3). New cstate: current_term: 1 leader_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } health_report { overall_health: HEALTHY } } }
Sep 26 16:37:06 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440624, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:06.624698 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.610237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.10.110.62:40149)
0926 16:37:06.610417 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.610423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.610950 (+ 527us) server_negotiation.cc:407] Connection header received
0926 16:37:06.611111 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.611114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.611207 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.611297 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:06.612469 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.613173 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.615027 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.615430 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.615530 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.619388 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:06.619392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:06.619396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:06.619425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:06.621906 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.622691 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.622694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.622697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.622813 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:06.623198 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:06.623201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:06.623203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:06.623463 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:06.623567 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.624106 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.624404 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":54}
I20260926 16:37:06.624758 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.610236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40149 (local address 127.0.0.1:42910)
0926 16:37:06.610840 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.610860 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.611004 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.611348 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.611352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.612077 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:06.612341 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.612349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.613323 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.613329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.614869 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.614883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.615600 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.615605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.615768 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.616443 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:06.616470 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:06.619204 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:06.622027 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:06.622039 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:06.622048 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:06.622579 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:06.622916 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:06.622921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:06.622925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:06.623084 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:06.623582 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:06.623591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:06.623791 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.623992 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.624294 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
I20260926 16:37:06.630679 11803 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42910:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:37:06.631956 11803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:06.639309 11940 tablet_service.cc:1467] Processing CreateTablet for tablet f00ec7ccb5b047daaa159b94316bfab8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=36a86b6c6bcd4744b3ad77bdc059c820]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:06.639550 12218 tablet_service.cc:1467] Processing CreateTablet for tablet f00ec7ccb5b047daaa159b94316bfab8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=36a86b6c6bcd4744b3ad77bdc059c820]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:06.640025 12218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00ec7ccb5b047daaa159b94316bfab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.640014 11940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00ec7ccb5b047daaa159b94316bfab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.644639 12078 tablet_service.cc:1467] Processing CreateTablet for tablet f00ec7ccb5b047daaa159b94316bfab8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=36a86b6c6bcd4744b3ad77bdc059c820]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:06.645182 12078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00ec7ccb5b047daaa159b94316bfab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:06.647692 12307 tablet_bootstrap.cc:492] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Bootstrap starting.
I20260926 16:37:06.649289 12307 tablet_bootstrap.cc:654] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.652371 12308 tablet_bootstrap.cc:492] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Bootstrap starting.
I20260926 16:37:06.653977 12308 tablet_bootstrap.cc:654] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.655709 12309 tablet_bootstrap.cc:492] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: Bootstrap starting.
I20260926 16:37:06.657653 12309 tablet_bootstrap.cc:654] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:06.664338 12307 tablet_bootstrap.cc:492] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: No bootstrap required, opened a new log
I20260926 16:37:06.664501 12307 ts_tablet_manager.cc:1397] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260926 16:37:06.664798 12308 tablet_bootstrap.cc:492] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: No bootstrap required, opened a new log
I20260926 16:37:06.664961 12308 ts_tablet_manager.cc:1397] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260926 16:37:06.665242 12307 raft_consensus.cc:348] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.665503 12307 raft_consensus.cc:374] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.665581 12307 raft_consensus.cc:729] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c34f761d8246fd9a3f7eb67a1c5f8b, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.665800 12309 tablet_bootstrap.cc:492] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: No bootstrap required, opened a new log
I20260926 16:37:06.665699 12308 raft_consensus.cc:348] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.665925 12308 raft_consensus.cc:374] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.665963 12309 ts_tablet_manager.cc:1397] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260926 16:37:06.666003 12308 raft_consensus.cc:729] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1dcd836a6c4cb0bac13c21bce07ad2, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.665856 12307 consensus_queue.cc:260] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.666265 12308 consensus_queue.cc:260] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.666272 12313 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.666530 12313 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.666664 12309 raft_consensus.cc:348] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.666679 12315 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.666890 12309 raft_consensus.cc:374] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:06.666956 12309 raft_consensus.cc:729] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9945c7937ad84c7e8b453da503f1d517, State: Initialized, Role: FOLLOWER
I20260926 16:37:06.666981 12315 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.667181 12309 consensus_queue.cc:260] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.667567 12314 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.667799 12314 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.668740 12309 ts_tablet_manager.cc:1428] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 16:37:06.669350 12307 ts_tablet_manager.cc:1428] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 16:37:06.669480 12308 ts_tablet_manager.cc:1428] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 16:37:06.732860 12315 raft_consensus.cc:482] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:06.733107 12315 raft_consensus.cc:504] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.734839 12315 leader_election.cc:288] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389), 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405)
I20260926 16:37:06.742730 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.735819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:34251 (local address 127.10.110.3:35405)
0926 16:37:06.736043 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.736048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.736380 (+ 332us) server_negotiation.cc:407] Connection header received
0926 16:37:06.736542 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.736545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.736600 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.736670 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.737255 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.738343 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.740823 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.741683 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.741839 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.741912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.742323 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.742458 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:37:06.742794 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.735144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:42023 (local address 127.10.110.1:33389)
0926 16:37:06.735357 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.735365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.735723 (+ 358us) server_negotiation.cc:407] Connection header received
0926 16:37:06.736179 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.736185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.736261 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.736320 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:06.737166 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.737948 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.740608 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.741897 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.742054 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.742140 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.742391 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.742546 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:37:06.742787 12350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.735695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:35405 (local address 127.10.110.2:34251)
0926 16:37:06.736288 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.736305 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.736422 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.736699 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.736703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.736724 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.737063 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.737071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.738469 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.738474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.740697 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.740710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.741869 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.741874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.742052 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.742183 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.742280 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.742407 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":22,"thread_start_us":343,"threads_started":1}
I20260926 16:37:06.743384 12238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00ec7ccb5b047daaa159b94316bfab8" candidate_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" is_pre_election: true
I20260926 16:37:06.743616 12238 raft_consensus.cc:2454] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1dcd836a6c4cb0bac13c21bce07ad2 in term 0.
I20260926 16:37:06.743749 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.735084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33389 (local address 127.10.110.2:42023)
0926 16:37:06.735629 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.735648 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.735849 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.736520 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.736525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.736551 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:06.736957 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.736968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.738061 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.738066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.740408 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.740423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.742092 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.742098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.742237 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.742265 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.743271 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.743409 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":51,"thread_start_us":273,"threads_started":1}
I20260926 16:37:06.744112 12031 leader_election.cc:302] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c34f761d8246fd9a3f7eb67a1c5f8b, 7b1dcd836a6c4cb0bac13c21bce07ad2; no voters:
I20260926 16:37:06.744516 12315 raft_consensus.cc:2781] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:06.744629 12315 raft_consensus.cc:482] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:06.744706 12315 raft_consensus.cc:3037] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.744903 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00ec7ccb5b047daaa159b94316bfab8" candidate_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9945c7937ad84c7e8b453da503f1d517" is_pre_election: true
I20260926 16:37:06.745128 11960 raft_consensus.cc:2454] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1dcd836a6c4cb0bac13c21bce07ad2 in term 0.
I20260926 16:37:06.747615 12315 raft_consensus.cc:504] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.748255 12315 leader_election.cc:288] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [CANDIDATE]: Term 1 election: Requested vote from peers 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389), 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405)
I20260926 16:37:06.748780 11960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00ec7ccb5b047daaa159b94316bfab8" candidate_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9945c7937ad84c7e8b453da503f1d517"
I20260926 16:37:06.748827 12238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00ec7ccb5b047daaa159b94316bfab8" candidate_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b"
I20260926 16:37:06.749009 12238 raft_consensus.cc:3037] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.749010 11960 raft_consensus.cc:3037] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:06.751788 12238 raft_consensus.cc:2454] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1dcd836a6c4cb0bac13c21bce07ad2 in term 1.
I20260926 16:37:06.751870 11960 raft_consensus.cc:2454] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1dcd836a6c4cb0bac13c21bce07ad2 in term 1.
I20260926 16:37:06.752359 12031 leader_election.cc:302] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1dcd836a6c4cb0bac13c21bce07ad2, 9945c7937ad84c7e8b453da503f1d517; no voters:
I20260926 16:37:06.752643 12315 raft_consensus.cc:2781] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:06.753118 12315 raft_consensus.cc:686] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 7b1dcd836a6c4cb0bac13c21bce07ad2, State: Running, Role: LEADER
I20260926 16:37:06.753510 12315 consensus_queue.cc:237] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } }
I20260926 16:37:06.755636 12351 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: TxnStatusTablet state changed. Reason: New leader 7b1dcd836a6c4cb0bac13c21bce07ad2. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.756115 12351 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:06.757473 11803 catalog_manager.cc:5495] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2). New cstate: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:06.762718 12353 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:37:06.762913 12353 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:37:06.792078 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.785082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.10.110.2:37659)
0926 16:37:06.785262 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.785267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.785315 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:37:06.785621 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.785626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.785729 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.785877 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:06.786257 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.787440 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.789364 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.790025 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.790171 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.790685 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.791411 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.791562 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.791649 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.791738 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:37:06.792466 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.784905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37659 (local address 127.0.0.1:42650)
0926 16:37:06.785235 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.785256 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.785406 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.785902 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.785907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.785919 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:06.786121 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.786132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.787589 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.787595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.789203 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.789218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.790186 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.790192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.790400 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.790474 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.791515 (+ 1041us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.791537 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.791629 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.791850 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 16:37:06.797165 12238 raft_consensus.cc:1261] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Refusing update from remote peer 7b1dcd836a6c4cb0bac13c21bce07ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:06.798034 11960 raft_consensus.cc:1261] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Refusing update from remote peer 7b1dcd836a6c4cb0bac13c21bce07ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:06.798513 12315 consensus_queue.cc:1035] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:06.798918 12351 consensus_queue.cc:1035] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:06.811733 12314 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: TxnStatusTablet state changed. Reason: New leader 7b1dcd836a6c4cb0bac13c21bce07ad2. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.812078 12314 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.816509 12315 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.816807 12315 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:06.816988 12322 mvcc.cc:205] Tried to move back new op lower bound from 7333644807352991744 to 7333644807186407424. Current Snapshot: MvccSnapshot[applied={T|T < 7333644807352991744}]
I20260926 16:37:06.821051 12321 mvcc.cc:205] Tried to move back new op lower bound from 7333644807352991744 to 7333644807186407424. Current Snapshot: MvccSnapshot[applied={T|T < 7333644807352991744 or (T in {7333644807352991744})}]
I20260926 16:37:06.821274 12351 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.821553 12351 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 7b1dcd836a6c4cb0bac13c21bce07ad2: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:06.821493 12314 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.821763 12314 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 9945c7937ad84c7e8b453da503f1d517: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.822606 12313 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: TxnStatusTablet state changed. Reason: New leader 7b1dcd836a6c4cb0bac13c21bce07ad2. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.822790 12323 mvcc.cc:205] Tried to move back new op lower bound from 7333644807352991744 to 7333644807186407424. Current Snapshot: MvccSnapshot[applied={T|T < 7333644807352991744}]
I20260926 16:37:06.822695 12338 tablet_replica.cc:439] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 } } peers { permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 } } peers { permanent_uuid: "17c34f761d8246fd9a3f7eb67a1c5f8b" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 35405 } } }
I20260926 16:37:06.822997 12338 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.822907 12313 tablet_replica.cc:442] T f00ec7ccb5b047daaa159b94316bfab8 P 17c34f761d8246fd9a3f7eb67a1c5f8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:06.837073 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.829165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.10.110.3:35405)
0926 16:37:06.829414 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.829420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.829440 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:37:06.829547 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.829552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.829647 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.829822 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:06.830245 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.831358 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.833306 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.834031 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.834152 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.834945 (+ 793us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.835867 (+ 922us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.836061 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.836682 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.836775 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 16:37:06.856782 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.849424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:53563 (local address 127.10.110.2:37659)
0926 16:37:06.849654 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.849660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.850264 (+ 604us) server_negotiation.cc:407] Connection header received
0926 16:37:06.850495 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.850500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.850586 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.850679 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:06.851241 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.852382 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.854386 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.855222 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.855343 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.855987 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.856129 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.856211 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.856394 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.856483 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:37:06.857627 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.849363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37659 (local address 127.10.110.3:53563)
0926 16:37:06.850147 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.850168 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.850334 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.850714 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.850719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.850732 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:06.851065 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.851078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.852522 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.852528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.854225 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.854240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.855428 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.855435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.855611 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.855728 (+ 117us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.856259 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.856273 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.857106 (+ 833us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.857255 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":459,"threads_started":1}
I20260926 16:37:06.876191 11960 raft_consensus.cc:1261] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Refusing update from remote peer 17c34f761d8246fd9a3f7eb67a1c5f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:06.876397 12104 raft_consensus.cc:1261] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Refusing update from remote peer 17c34f761d8246fd9a3f7eb67a1c5f8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:06.876833 12338 consensus_queue.cc:1035] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9945c7937ad84c7e8b453da503f1d517" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:06.877470 12338 consensus_queue.cc:1035] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b1dcd836a6c4cb0bac13c21bce07ad2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:06.955734 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.944700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:34249 (local address 127.10.110.3:35405)
0926 16:37:06.945015 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:06.945022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:06.948519 (+ 3497us) server_negotiation.cc:407] Connection header received
0926 16:37:06.948710 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:06.948715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:06.948804 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:06.948906 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:06.949722 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.950823 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.953433 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.954151 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.954288 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.954810 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.954966 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.955027 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:06.955272 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 16:37:06.955406 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260926 16:37:06.957139 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:06.944543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:35405 (local address 127.10.110.2:34249)
0926 16:37:06.948397 (+ 3854us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:06.948417 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:06.948570 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:06.949115 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:06.949121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:06.949138 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:06.949519 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.949532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.951390 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.951395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.953268 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:06.953283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:06.954329 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:06.954336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:06.954505 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:06.954582 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:06.955127 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:06.955138 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:37:06.956040 (+ 902us) client_negotiation.cc:240] Negotiation successful
0926 16:37:06.956634 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3646,"thread_start_us":3493,"threads_started":1}
I20260926 16:37:07.010093 11801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:07.010404 11801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:07.014045 11801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f43107f7089c47458b532ce5468e45ce: Sending DeleteTablet for 3 replicas of tablet 8e4c4ec30430491b9341f2b0e36034e9
I20260926 16:37:07.014961 11940 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4c4ec30430491b9341f2b0e36034e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:07 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:36038
I20260926 16:37:07.015259 12332 tablet_replica.cc:330] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: stopping tablet replica
I20260926 16:37:07.015475 12332 raft_consensus.cc:2229] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:07.015600 12078 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4c4ec30430491b9341f2b0e36034e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:07 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42638
I20260926 16:37:07.015708 12332 raft_consensus.cc:2258] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:07.015995 12333 tablet_replica.cc:330] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: stopping tablet replica
I20260926 16:37:07.016232 12333 raft_consensus.cc:2229] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:07.016433 12333 raft_consensus.cc:2258] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:07.016398 12218 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4c4ec30430491b9341f2b0e36034e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:07 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:48926
I20260926 16:37:07.016630 12331 tablet_replica.cc:330] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: stopping tablet replica
I20260926 16:37:07.016817 12331 raft_consensus.cc:2229] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:07.017349 12331 raft_consensus.cc:2258] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:07.017607 12333 ts_tablet_manager.cc:1905] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:07.018450 12331 ts_tablet_manager.cc:1905] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:07.024946 12332 ts_tablet_manager.cc:1905] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:07.025518 12331 ts_tablet_manager.cc:1918] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:07.025648 12331 log.cc:1196] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/8e4c4ec30430491b9341f2b0e36034e9
I20260926 16:37:07.026039 12331 ts_tablet_manager.cc:1939] T 8e4c4ec30430491b9341f2b0e36034e9 P 17c34f761d8246fd9a3f7eb67a1c5f8b: Deleting consensus metadata
I20260926 16:37:07.027113 11788 catalog_manager.cc:4841] TS 17c34f761d8246fd9a3f7eb67a1c5f8b (127.10.110.3:35405): tablet 8e4c4ec30430491b9341f2b0e36034e9 (table test-table [id=bd3d87192ca14f0c89a9a83bf3c35628]) successfully deleted
I20260926 16:37:07.033383 12332 ts_tablet_manager.cc:1918] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:07.035002 12333 ts_tablet_manager.cc:1918] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:07.043969 12333 log.cc:1196] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/8e4c4ec30430491b9341f2b0e36034e9
I20260926 16:37:07.044562 12333 ts_tablet_manager.cc:1939] T 8e4c4ec30430491b9341f2b0e36034e9 P 7b1dcd836a6c4cb0bac13c21bce07ad2: Deleting consensus metadata
I20260926 16:37:07.045073 12332 log.cc:1196] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/8e4c4ec30430491b9341f2b0e36034e9
I20260926 16:37:07.045475 12332 ts_tablet_manager.cc:1939] T 8e4c4ec30430491b9341f2b0e36034e9 P 9945c7937ad84c7e8b453da503f1d517: Deleting consensus metadata
I20260926 16:37:07.046459 11788 catalog_manager.cc:4841] TS 9945c7937ad84c7e8b453da503f1d517 (127.10.110.1:33389): tablet 8e4c4ec30430491b9341f2b0e36034e9 (table test-table [id=bd3d87192ca14f0c89a9a83bf3c35628]) successfully deleted
I20260926 16:37:07.047204 11790 catalog_manager.cc:4841] TS 7b1dcd836a6c4cb0bac13c21bce07ad2 (127.10.110.2:37659): tablet 8e4c4ec30430491b9341f2b0e36034e9 (table test-table [id=bd3d87192ca14f0c89a9a83bf3c35628]) successfully deleted
Sep 26 16:37:07 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:07.059198 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:07.037109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.10.110.1:33389)
0926 16:37:07.037280 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:07.037286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:07.037869 (+ 583us) server_negotiation.cc:407] Connection header received
0926 16:37:07.038131 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:07.038138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:07.038214 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:07.038274 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:07.040377 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.041502 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.043365 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.044014 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.044122 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:07.052409 (+ 8287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:07.052425 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:07.052445 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:07.052498 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:07.056141 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.056965 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.056975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.056979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.057142 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.057552 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.057557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.057561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.057998 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:07.058303 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:07.058622 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 16:37:07.058851 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 16:37:07.059916 12005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36050
I20260926 16:37:07.079098 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:07.063547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.10.110.62:40149)
0926 16:37:07.063726 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:07.063731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:07.064181 (+ 450us) server_negotiation.cc:407] Connection header received
0926 16:37:07.064303 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:07.064308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:07.064390 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:07.064488 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:07.066032 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.067180 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.068999 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.069652 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.069792 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:07.072674 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:07.072681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:07.072686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:07.072729 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:07.076233 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.077021 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.077027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.077031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.077178 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.077568 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.077573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.077576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.077914 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:07.078205 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:07.078516 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 16:37:07.078743 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:37:07.079723 11801 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42926
I20260926 16:37:07.081691 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11874
I20260926 16:37:07.098873 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:07.083299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.10.110.1:33389)
0926 16:37:07.083470 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:07.083475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:07.084307 (+ 832us) server_negotiation.cc:407] Connection header received
0926 16:37:07.084483 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:07.084489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:07.084570 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:07.084663 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:07.086025 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.087150 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.088905 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.089567 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.089705 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:07.092568 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:07.092575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:07.092579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:07.092645 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:07.095910 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.096681 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.096687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.096690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.096843 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.097245 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.097250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.097253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.097581 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:07.097820 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:07.098121 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:37:07.098346 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:37:07.099418 12005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36064
W20260926 16:37:07.100056 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11874: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:07.100250 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11874
I20260926 16:37:07.129066 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12014
Sep 26 16:37:07 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:07.149192 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:07.129891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.10.110.2:37659)
0926 16:37:07.130092 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:07.130099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:07.130258 (+ 159us) server_negotiation.cc:407] Connection header received
0926 16:37:07.130427 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:07.130432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:07.130517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:07.130635 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:07.132274 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.133429 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.135274 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.135962 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.136072 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:07.141515 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:07.141531 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:07.141551 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:07.141600 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:07.145633 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.146533 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.146543 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.146546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.146739 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.147208 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.147212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.147216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.147647 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:07.147959 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:07.148539 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 16:37:07.148766 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 16:37:07.150780 12144 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42666
W20260926 16:37:07.151518 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:07.151682 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 12014
I20260926 16:37:07.167501 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12153
Sep 26 16:37:07 dist-test-slave-whjp krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:07.181821 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:07.168200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.10.110.3:35405)
0926 16:37:07.168373 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:07.168379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:07.168488 (+ 109us) server_negotiation.cc:407] Connection header received
0926 16:37:07.168686 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:07.168691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:07.168758 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:07.168838 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:07.170002 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.170707 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.171940 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:07.172352 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:07.172459 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:07.176345 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:07.176357 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:07.176372 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:07.176409 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:07.179181 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.179777 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.179784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.179786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.180049 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:07.180385 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:07.180390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:07.180394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:07.180735 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:07.181052 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:07.181274 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 16:37:07.181475 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 16:37:07.182471 12283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48934
W20260926 16:37:07.183133 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12153: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:07.183296 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 12153
I20260926 16:37:07.197443 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 11771
I20260926 16:37:07.198490 11843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:07.309823 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 11771
2026-09-26T16:37:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4339 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:07 dist-test-slave-whjp krb5kdc[12377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:07 dist-test-slave-whjp krb5kdc[12377](info): set up 2 sockets
Sep 26 16:37:07 dist-test-slave-whjp krb5kdc[12377](info): commencing operation
krb5kdc: starting...
W20260926 16:37:08.292500 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:08 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:08.465291 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:32813
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:08.774273 12390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:08.774549 12390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:08.774633 12390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:08.781757 12390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:08.781859 12390 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:37:08.781896 12390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:08.781952 12390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:08.782006 12390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:08.782056 12390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:08.791740 12390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:32813
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:08.792965 12390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:08.794219 12390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:08.805157 12396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:08.805234 12395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:08.805217 12398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:08.806177 12390 server_base.cc:1034] running on GCE node
I20260926 16:37:08.806891 12390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:08.808318 12390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:08.809544 12390 hybrid_clock.cc:648] HybridClock initialized: now 1790440628809496 us; error 38 us; skew 500 ppm
I20260926 16:37:08.809880 12390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:08 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:08.812778 12390 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:08.814182 12390 webserver.cc:460] Webserver started at http://127.10.110.62:42779/ using document root <none> and password file <none>
I20260926 16:37:08.814853 12390 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:08.814956 12390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:08.815246 12390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:08.818100 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "37832cb179a3465084c91db58e56ecbf"
format_stamp: "Formatted at 2026-09-26 16:37:08 on dist-test-slave-whjp"
server_key: "aeb5d15ba8f981e0cc99921495ad798f"
server_key_iv: "d31164d38ae7c777ea9e6aebb308d3c6"
server_key_version: "encryptionkey@0"
I20260926 16:37:08.818820 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "37832cb179a3465084c91db58e56ecbf"
format_stamp: "Formatted at 2026-09-26 16:37:08 on dist-test-slave-whjp"
server_key: "aeb5d15ba8f981e0cc99921495ad798f"
server_key_iv: "d31164d38ae7c777ea9e6aebb308d3c6"
server_key_version: "encryptionkey@0"
I20260926 16:37:08.823701 12390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:37:08.827082 12405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:08.828454 12390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:08.828606 12390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "37832cb179a3465084c91db58e56ecbf"
format_stamp: "Formatted at 2026-09-26 16:37:08 on dist-test-slave-whjp"
server_key: "aeb5d15ba8f981e0cc99921495ad798f"
server_key_iv: "d31164d38ae7c777ea9e6aebb308d3c6"
server_key_version: "encryptionkey@0"
I20260926 16:37:08.828799 12390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:08.845439 12390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:08.849248 12390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:08.849471 12390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:08.870636 12390 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:32813
I20260926 16:37:08.870648 12466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:32813 every 8 connection(s)
I20260926 16:37:08.872125 12390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:08.875859 12467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:08.881338 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 12390
I20260926 16:37:08.881506 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:08.882099 10680 external_mini_cluster.cc:1428] Setting key 849ffb7182d3abcae6b3b83ebf8753a5
I20260926 16:37:08.886962 12467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Bootstrap starting.
I20260926 16:37:08.890672 12467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:08.891783 12467 log.cc:824] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:08.894706 12467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: No bootstrap required, opened a new log
Sep 26 16:37:08 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:08.904086 12467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37832cb179a3465084c91db58e56ecbf" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32813 } }
I20260926 16:37:08.904533 12467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:08.904635 12467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37832cb179a3465084c91db58e56ecbf, State: Initialized, Role: FOLLOWER
I20260926 16:37:08.905330 12467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37832cb179a3465084c91db58e56ecbf" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32813 } }
I20260926 16:37:08.905570 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:08.884058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.10.110.62:32813)
0926 16:37:08.884869 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:08.884885 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:08.884934 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:37:08.887099 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:08.887127 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:08.887632 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:08.888064 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:08.889281 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:08.891205 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:08.892620 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:08.893467 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:08.893652 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:08.897261 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:08.897271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:08.897290 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:08.897342 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:08.901193 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:08.901849 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:08.901857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:08.901867 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:08.902085 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:08.902402 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:08.902407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:08.902410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:08.903136 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:08.903402 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:08.903734 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 16:37:08.904184 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":343,"threads_started":1}
I20260926 16:37:08.905618 12467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:08.905732 12467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:08.905911 12467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:08.908160 12467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37832cb179a3465084c91db58e56ecbf" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32813 } }
I20260926 16:37:08.908622 12467 leader_election.cc:302] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37832cb179a3465084c91db58e56ecbf; no voters:
I20260926 16:37:08.909112 12467 leader_election.cc:288] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:08.909228 12472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:08.910436 12472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [term 1 LEADER]: Becoming Leader. State: Replica: 37832cb179a3465084c91db58e56ecbf, State: Running, Role: LEADER
I20260926 16:37:08.910955 12472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37832cb179a3465084c91db58e56ecbf" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32813 } }
I20260926 16:37:08.911089 12467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:08.913646 12473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37832cb179a3465084c91db58e56ecbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37832cb179a3465084c91db58e56ecbf" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32813 } } }
I20260926 16:37:08.914000 12473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:08.913614 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37832cb179a3465084c91db58e56ecbf. Latest consensus state: current_term: 1 leader_uuid: "37832cb179a3465084c91db58e56ecbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37832cb179a3465084c91db58e56ecbf" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32813 } } }
I20260926 16:37:08.914321 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:08.915575 12478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:08.920523 12478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:08.929384 12478 catalog_manager.cc:1347] Generated new cluster ID: 35ad410cf06146399194607c964eb4e4
I20260926 16:37:08.929515 12478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:08.943718 12478 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:08.945024 12478 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:08.953004 12478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Generated new TSK 0
I20260926 16:37:08.953830 12478 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:08.999356 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32813
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:09.331324 12493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:09.331601 12493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:09.331744 12493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:09.338917 12493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:09.339025 12493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:09.339108 12493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:37:09.339159 12493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:09.339218 12493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:09.348888 12493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:09.350171 12493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:09.351395 12493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:09.363845 12498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:09.363773 12499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:09.363799 12501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:09.364686 12493 server_base.cc:1034] running on GCE node
I20260926 16:37:09.365232 12493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:09.366096 12493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:09.367281 12493 hybrid_clock.cc:648] HybridClock initialized: now 1790440629367225 us; error 47 us; skew 500 ppm
I20260926 16:37:09.367578 12493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:09 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:09.370616 12493 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:09.371994 12493 webserver.cc:460] Webserver started at http://127.10.110.1:36941/ using document root <none> and password file <none>
I20260926 16:37:09.372648 12493 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:09.372763 12493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:09.373037 12493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:09.375803 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "3a5e78fd3f084adc96b78a863b76234f"
format_stamp: "Formatted at 2026-09-26 16:37:09 on dist-test-slave-whjp"
server_key: "389f45a0548babaf331c58ae85bf64fe"
server_key_iv: "691d090ba3027b64a6d063407a78c1dd"
server_key_version: "encryptionkey@0"
I20260926 16:37:09.376598 12493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "3a5e78fd3f084adc96b78a863b76234f"
format_stamp: "Formatted at 2026-09-26 16:37:09 on dist-test-slave-whjp"
server_key: "389f45a0548babaf331c58ae85bf64fe"
server_key_iv: "691d090ba3027b64a6d063407a78c1dd"
server_key_version: "encryptionkey@0"
I20260926 16:37:09.381567 12493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:37:09.384840 12508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:09.386121 12493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:09.386269 12493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "3a5e78fd3f084adc96b78a863b76234f"
format_stamp: "Formatted at 2026-09-26 16:37:09 on dist-test-slave-whjp"
server_key: "389f45a0548babaf331c58ae85bf64fe"
server_key_iv: "691d090ba3027b64a6d063407a78c1dd"
server_key_version: "encryptionkey@0"
I20260926 16:37:09.386430 12493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:09.404927 12493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:09.408990 12493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:09.409253 12493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:09.411868 12493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:09.411973 12493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:09.412082 12493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:09.412160 12493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:09 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:09.435717 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.412600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48251 (local address 127.10.110.62:32813)
0926 16:37:09.413110 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:09.413116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:09.414307 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 16:37:09.417094 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:09.417101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:09.417195 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:09.417320 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:09.419792 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.420753 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.423490 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.424090 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.424222 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.429223 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:09.429228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:09.429231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:09.429265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:09.431683 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.432702 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.432708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.432711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.432871 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.433549 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.433554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.433558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.433906 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:09.434055 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:09.435151 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0926 16:37:09.435373 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":287,"threads_started":1}
I20260926 16:37:09.437188 12518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.413269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.10.110.1:48251)
0926 16:37:09.414128 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:09.414179 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:09.416729 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:09.417607 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:09.417618 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:09.418524 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:09.419566 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.419598 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.420930 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.420936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.423333 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.423351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.424531 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.424539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.424832 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.425679 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:09.425719 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:09.428967 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:09.431852 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.431861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.431883 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.432555 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.433061 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.433066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.433070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.433386 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.434117 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:09.434124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:09.434640 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 16:37:09.434996 (+ 356us) client_negotiation.cc:240] Negotiation successful
0926 16:37:09.435544 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":357,"threads_started":1}
I20260926 16:37:09.459607 12493 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:34903
I20260926 16:37:09.459625 12628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:34903 every 8 connection(s)
I20260926 16:37:09.460886 12493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:09.461925 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.448039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:60529 (local address 127.10.110.62:32813)
0926 16:37:09.448220 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:09.448226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:09.449045 (+ 819us) server_negotiation.cc:407] Connection header received
0926 16:37:09.449220 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:09.449225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:09.449309 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:09.449429 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:09.450787 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.451888 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.453549 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.454245 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.454377 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.456575 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:09.456581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:09.456584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:09.456615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:09.459298 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.460075 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.460081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.460085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.460218 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.460552 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.460561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.460564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.460819 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:09.461013 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:09.461371 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:37:09.461589 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:37:09.462163 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.448114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.10.110.1:60529)
0926 16:37:09.448924 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:09.448946 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:09.449106 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:09.449510 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:09.449515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:09.450140 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:09.450569 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.450580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.452040 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.452044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.453404 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.453418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.454431 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.454435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.454543 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.455211 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:09.455240 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:09.456322 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:09.459425 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.459431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.459435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.459966 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.460321 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.460327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.460331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.460448 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.460931 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:09.460936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:09.461135 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 16:37:09.461250 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:37:09.461702 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":525,"threads_started":1}
I20260926 16:37:09.463413 12629 heartbeater.cc:347] Connected to a master server at 127.10.110.62:32813
I20260926 16:37:09.463748 12629 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:09.464612 12629 heartbeater.cc:511] Master 127.10.110.62:32813 requested a full tablet report, sending...
I20260926 16:37:09.466588 12421 ts_manager.cc:194] Registered new tserver with Master: 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903)
I20260926 16:37:09.467995 12421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:48251
I20260926 16:37:09.469108 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 12493
I20260926 16:37:09.469287 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:09.470052 10680 external_mini_cluster.cc:1428] Setting key 12b56f8a7ea1818519367284af954ed4
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:09.506315 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32813
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:09.817974 12633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:09.818293 12633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:09.818409 12633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:09.825799 12633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:09.825920 12633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:09.826028 12633 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:37:09.826085 12633 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:09.826161 12633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:09.836061 12633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:09.837319 12633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:09.838634 12633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:09.851763 12638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:09.851734 12641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:09.851704 12639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:09.851931 12633 server_base.cc:1034] running on GCE node
I20260926 16:37:09.852936 12633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:09.853896 12633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:09.855094 12633 hybrid_clock.cc:648] HybridClock initialized: now 1790440629855040 us; error 52 us; skew 500 ppm
I20260926 16:37:09.855388 12633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:09 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:09.858362 12633 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:09.859709 12633 webserver.cc:460] Webserver started at http://127.10.110.2:38433/ using document root <none> and password file <none>
I20260926 16:37:09.860370 12633 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:09.860466 12633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:09.860713 12633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:09.863489 12633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "b362bac792784b3ca2511e75dcdb07eb"
format_stamp: "Formatted at 2026-09-26 16:37:09 on dist-test-slave-whjp"
server_key: "d0e667df8663c55e6dfa999067c211f7"
server_key_iv: "c6658484a62e0c5debdbce310d29901a"
server_key_version: "encryptionkey@0"
I20260926 16:37:09.864221 12633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "b362bac792784b3ca2511e75dcdb07eb"
format_stamp: "Formatted at 2026-09-26 16:37:09 on dist-test-slave-whjp"
server_key: "d0e667df8663c55e6dfa999067c211f7"
server_key_iv: "c6658484a62e0c5debdbce310d29901a"
server_key_version: "encryptionkey@0"
I20260926 16:37:09.869230 12633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:37:09.872757 12648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:09.874083 12633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:09.874210 12633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "b362bac792784b3ca2511e75dcdb07eb"
format_stamp: "Formatted at 2026-09-26 16:37:09 on dist-test-slave-whjp"
server_key: "d0e667df8663c55e6dfa999067c211f7"
server_key_iv: "c6658484a62e0c5debdbce310d29901a"
server_key_version: "encryptionkey@0"
I20260926 16:37:09.874377 12633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:09.889485 12633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:09.893731 12633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:09.893985 12633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:09.896519 12633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:09.896611 12633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:09.896715 12633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:09.896780 12633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:09 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:09.918889 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.897156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:54737 (local address 127.10.110.62:32813)
0926 16:37:09.897335 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:09.897340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:09.898742 (+ 1402us) server_negotiation.cc:407] Connection header received
0926 16:37:09.901151 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:09.901157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:09.901279 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:09.901345 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:09.903000 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.904168 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.906223 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.906854 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.906988 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.911681 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:09.911688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:09.911693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:09.911737 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:09.915117 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.916068 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.916074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.916078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.916231 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.916834 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.916839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.916842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.917230 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:09.917419 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:09.918323 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 16:37:09.918545 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:37:09.920385 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.897731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.10.110.2:54737)
0926 16:37:09.898547 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:09.898609 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:09.900813 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:09.901751 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:09.901759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:09.902251 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:09.902822 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.902840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.904340 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.904346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.906070 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.906083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.907097 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.907109 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.907318 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.908096 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:09.908130 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:09.911350 (+ 3220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:09.915276 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.915286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.915305 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.915942 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.916344 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.916349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.916353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.916659 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.917403 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:09.917412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:09.917854 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 16:37:09.918229 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 16:37:09.918814 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":343,"threads_started":1}
I20260926 16:37:09.935750 12633 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:37263
I20260926 16:37:09.935871 12767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:37263 every 8 connection(s)
I20260926 16:37:09.936990 12633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:09.939253 12768 heartbeater.cc:347] Connected to a master server at 127.10.110.62:32813
I20260926 16:37:09.939464 12768 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:09.940325 12768 heartbeater.cc:511] Master 127.10.110.62:32813 requested a full tablet report, sending...
I20260926 16:37:09.941547 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 12633
I20260926 16:37:09.941715 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:09.941933 12421 ts_manager.cc:194] Registered new tserver with Master: b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:09.942236 10680 external_mini_cluster.cc:1428] Setting key facc4df5ac49ef7447d0b3ba4de83bdd
I20260926 16:37:09.942739 12421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:54737
I20260926 16:37:09.952347 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.928049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:47817 (local address 127.10.110.62:32813)
0926 16:37:09.928230 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:09.928237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:09.940475 (+ 12238us) server_negotiation.cc:407] Connection header received
0926 16:37:09.940642 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:09.940647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:09.940728 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:09.940832 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:09.941784 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.942927 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.944681 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.945273 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.945393 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.947363 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:09.947368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:09.947372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:09.947404 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:09.949911 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.950499 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.950505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.950509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.950645 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:09.950974 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:09.950979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:09.950982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:09.951302 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:09.951429 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:09.951802 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 16:37:09.952020 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:37:09.952416 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:09.927804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.10.110.2:47817)
0926 16:37:09.940374 (+ 12570us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:09.940394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:09.940534 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:09.940841 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:09.940844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:09.941380 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:09.941653 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.941666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.943066 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.943071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.944461 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:09.944477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:09.945469 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:09.945476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:09.945671 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:09.946249 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:09.946270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:09.947176 (+ 906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:09.950031 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.950036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.950040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.950396 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.950746 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:09.950751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:09.950755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:09.950874 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:09.951410 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:09.951414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:09.951578 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 16:37:09.951694 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:37:09.952002 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12357,"thread_start_us":12193,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:09.982501 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32813
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:10.291378 12772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:10.291651 12772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:10.291785 12772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:10.299012 12772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:10.299137 12772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:10.299260 12772 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:37:10.299322 12772 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:10.299425 12772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:10.309177 12772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:32813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:10.310695 12772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:10.312070 12772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:10.324649 12778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:10.324772 12780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:10.324762 12777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:10.326112 12772 server_base.cc:1034] running on GCE node
I20260926 16:37:10.326692 12772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:10.327638 12772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:10.328833 12772 hybrid_clock.cc:648] HybridClock initialized: now 1790440630328777 us; error 46 us; skew 500 ppm
I20260926 16:37:10.329149 12772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:10 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:10.332301 12772 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:10.333814 12772 webserver.cc:460] Webserver started at http://127.10.110.3:36805/ using document root <none> and password file <none>
I20260926 16:37:10.334472 12772 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:10.334573 12772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:10.334874 12772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:10.337813 12772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "e8e4c6c3b79a4eb4880f44415ef934fb"
format_stamp: "Formatted at 2026-09-26 16:37:10 on dist-test-slave-whjp"
server_key: "7dcb2501ef4ae45da160a7a7b9d3d486"
server_key_iv: "e64a6ea0f5aaf19f5ec668a4ca121175"
server_key_version: "encryptionkey@0"
I20260926 16:37:10.338538 12772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "e8e4c6c3b79a4eb4880f44415ef934fb"
format_stamp: "Formatted at 2026-09-26 16:37:10 on dist-test-slave-whjp"
server_key: "7dcb2501ef4ae45da160a7a7b9d3d486"
server_key_iv: "e64a6ea0f5aaf19f5ec668a4ca121175"
server_key_version: "encryptionkey@0"
I20260926 16:37:10.343631 12772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:37:10.347157 12787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:10.348570 12772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:10.348706 12772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "e8e4c6c3b79a4eb4880f44415ef934fb"
format_stamp: "Formatted at 2026-09-26 16:37:10 on dist-test-slave-whjp"
server_key: "7dcb2501ef4ae45da160a7a7b9d3d486"
server_key_iv: "e64a6ea0f5aaf19f5ec668a4ca121175"
server_key_version: "encryptionkey@0"
I20260926 16:37:10.348896 12772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:10.365317 12772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:10.369381 12772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:10.369688 12772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:10.373984 12772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:10.374152 12772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:37:10.374285 12772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:10.374383 12772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:10 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440630, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:10.397852 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.374075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:58449 (local address 127.10.110.62:32813)
0926 16:37:10.374293 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.374325 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.377036 (+ 2711us) server_negotiation.cc:407] Connection header received
0926 16:37:10.379967 (+ 2931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.379973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.380072 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.380321 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:10.382702 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.383685 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.385963 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.386471 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.386607 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.391296 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:10.391301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:10.391305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:10.391337 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:10.394209 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.395203 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.395208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.395212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.395359 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.395797 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.395802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.395805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.396134 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:10.396282 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.397259 (+ 977us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.397487 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:37:10.399333 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.374384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.10.110.3:58449)
0926 16:37:10.376853 (+ 2469us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.376908 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.379597 (+ 2689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.380630 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.380646 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.381601 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:10.382492 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.382533 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.383917 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.383922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.385808 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.385823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.386887 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.386895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.387193 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.388127 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:10.388187 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:10.391085 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:10.394365 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:10.394375 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:10.394398 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:10.395040 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:10.395468 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:10.395473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:10.395477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:10.395679 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:10.396362 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:10.396371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:10.396816 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.397142 (+ 326us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.397713 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2156,"thread_start_us":1929,"threads_started":1}
I20260926 16:37:10.423956 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.410303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.10.110.3:59835)
0926 16:37:10.411140 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.411156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.411269 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.411634 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.411639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.412182 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:10.412707 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.412722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.413775 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.413780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.415237 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.415254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.416251 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.416258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.416376 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.416939 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:10.416956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:10.417785 (+ 829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:10.421365 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:10.421371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:10.421378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:10.421882 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:10.422308 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:10.422312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:10.422315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:10.422443 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:10.422989 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:10.422993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:10.423217 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.423328 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.423593 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":537,"threads_started":1}
I20260926 16:37:10.424072 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.410292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:59835 (local address 127.10.110.62:32813)
0926 16:37:10.410479 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.410486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.411227 (+ 741us) server_negotiation.cc:407] Connection header received
0926 16:37:10.411380 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.411385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.411469 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.411576 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:10.412840 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.413651 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.415381 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.416051 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.416180 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.417973 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:10.417980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:10.417985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:10.418021 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:10.421227 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.422009 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.422015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.422020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.422178 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.422530 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.422535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.422538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.422892 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:10.423047 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.423446 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.423694 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:37:10.430197 12772 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:43553
I20260926 16:37:10.430225 12906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:43553 every 8 connection(s)
I20260926 16:37:10.431341 12772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:10.433373 12907 heartbeater.cc:347] Connected to a master server at 127.10.110.62:32813
I20260926 16:37:10.433593 12907 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:10.434197 12907 heartbeater.cc:511] Master 127.10.110.62:32813 requested a full tablet report, sending...
I20260926 16:37:10.435464 12421 ts_manager.cc:194] Registered new tserver with Master: e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553)
I20260926 16:37:10.436307 12421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:58449
I20260926 16:37:10.439244 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 12772
I20260926 16:37:10.439393 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:10.439800 10680 external_mini_cluster.cc:1428] Setting key 57e10f2bc560ce778b4a8d8d93f9feac
I20260926 16:37:10.442710 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:37:10 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260926 16:37:10.471310 12629 heartbeater.cc:503] Master 127.10.110.62:32813 was elected leader, sending a full tablet report...
Sep 26 16:37:10 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:10.485179 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.471990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.10.110.62:32813)
0926 16:37:10.472155 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.472160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.472499 (+ 339us) server_negotiation.cc:407] Connection header received
0926 16:37:10.472631 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.472636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.472728 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.472822 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:10.474192 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.474917 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.476159 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.476560 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.476673 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.480466 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:10.480474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:10.480479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:10.480517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:10.482739 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.483329 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.483334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.483336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.483456 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.483802 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.483805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.483807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.484122 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:10.484340 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.484641 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.484856 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:37:10.489060 12421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:10.495637 12421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:10.530620 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.508534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.10.110.2:37263)
0926 16:37:10.523811 (+ 15277us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.523819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.524421 (+ 602us) server_negotiation.cc:407] Connection header received
0926 16:37:10.524569 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.524577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.524823 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.525005 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:10.525542 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.526574 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.528485 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.529525 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.529647 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.529872 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.530154 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.530323 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15122,"thread_start_us":14990,"threads_started":1}
I20260926 16:37:10.530706 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.508071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37263 (local address 127.0.0.1:35428)
0926 16:37:10.524313 (+ 16242us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.524329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.524438 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.525006 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.525011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.525034 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:10.525378 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.525388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.526696 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.526701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.528318 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.528331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.529701 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.529722 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.529906 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.529953 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.530087 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.530299 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16129,"mutex_wait_us":1,"thread_start_us":15961,"threads_started":1}
I20260926 16:37:10.531129 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.506731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:43553 (local address 127.0.0.1:44308)
0926 16:37:10.521527 (+ 14796us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.521585 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.521779 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.523623 (+ 1844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.523628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.523651 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:10.524213 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.524226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.525996 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.526001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.528103 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.528116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.530125 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.530132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.530386 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.530419 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.530496 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.530616 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14487,"mutex_wait_us":72,"thread_start_us":15326,"threads_started":1}
I20260926 16:37:10.531286 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.506840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.10.110.3:43553)
0926 16:37:10.522703 (+ 15863us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.522712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.522746 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:37:10.522839 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.522847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.523098 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.523328 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:10.524396 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.525857 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.528283 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.529596 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.529733 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.529944 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.530837 (+ 893us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.530982 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15178,"thread_start_us":15034,"threads_started":1}
I20260926 16:37:10.532727 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.507194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53426 (local address 127.10.110.1:34903)
0926 16:37:10.521007 (+ 13813us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.521017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.522476 (+ 1459us) server_negotiation.cc:407] Connection header received
0926 16:37:10.522702 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.522710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.522996 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.523205 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:10.524084 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.525388 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.528174 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.529596 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.529725 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.529919 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.530543 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.530711 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13638,"thread_start_us":13506,"threads_started":1}
I20260926 16:37:10.533346 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.506756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34903 (local address 127.0.0.1:53426)
0926 16:37:10.522374 (+ 15618us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.522387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.522502 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.523227 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.523234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.523261 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:10.523819 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.523854 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.525546 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.525551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.527801 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.527815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.530097 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.530103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.530272 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.530375 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.532955 (+ 2580us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.533081 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15454,"mutex_wait_us":30,"thread_start_us":14315,"threads_started":1}
I20260926 16:37:10.534667 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 85010dbcea8440409d9696e05e477083 (DEFAULT_TABLE table=test-table [id=164a9fe8f1f648d696c5711a17565fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:10.535063 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 85010dbcea8440409d9696e05e477083 (DEFAULT_TABLE table=test-table [id=164a9fe8f1f648d696c5711a17565fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:10.536203 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85010dbcea8440409d9696e05e477083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:10.536296 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85010dbcea8440409d9696e05e477083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:10.541592 12559 tablet_service.cc:1467] Processing CreateTablet for tablet 85010dbcea8440409d9696e05e477083 (DEFAULT_TABLE table=test-table [id=164a9fe8f1f648d696c5711a17565fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:10.542691 12559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85010dbcea8440409d9696e05e477083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:10.550343 12926 tablet_bootstrap.cc:492] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Bootstrap starting.
I20260926 16:37:10.551048 12927 tablet_bootstrap.cc:492] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Bootstrap starting.
I20260926 16:37:10.554554 12928 tablet_bootstrap.cc:492] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Bootstrap starting.
I20260926 16:37:10.554706 12926 tablet_bootstrap.cc:654] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:10.555124 12927 tablet_bootstrap.cc:654] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:10.556027 12926 log.cc:824] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:10.556394 12927 log.cc:824] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:10.557792 12928 tablet_bootstrap.cc:654] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:10.559003 12926 tablet_bootstrap.cc:492] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: No bootstrap required, opened a new log
I20260926 16:37:10.558920 12928 log.cc:824] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:10.559268 12926 ts_tablet_manager.cc:1397] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:37:10.559360 12927 tablet_bootstrap.cc:492] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: No bootstrap required, opened a new log
I20260926 16:37:10.559767 12927 ts_tablet_manager.cc:1397] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 16:37:10.562170 12928 tablet_bootstrap.cc:492] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: No bootstrap required, opened a new log
I20260926 16:37:10.562417 12928 ts_tablet_manager.cc:1397] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 16:37:10.566524 12926 raft_consensus.cc:348] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.566895 12926 raft_consensus.cc:374] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:10.567008 12926 raft_consensus.cc:729] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Initialized, Role: FOLLOWER
I20260926 16:37:10.567579 12926 consensus_queue.cc:260] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.569963 12907 heartbeater.cc:503] Master 127.10.110.62:32813 was elected leader, sending a full tablet report...
I20260926 16:37:10.569747 12927 raft_consensus.cc:348] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.570164 12927 raft_consensus.cc:374] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:10.570245 12927 raft_consensus.cc:729] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b362bac792784b3ca2511e75dcdb07eb, State: Initialized, Role: FOLLOWER
I20260926 16:37:10.570820 12926 ts_tablet_manager.cc:1428] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260926 16:37:10.570886 12927 consensus_queue.cc:260] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.572026 12768 heartbeater.cc:503] Master 127.10.110.62:32813 was elected leader, sending a full tablet report...
I20260926 16:37:10.572285 12928 raft_consensus.cc:348] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.572605 12927 ts_tablet_manager.cc:1428] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260926 16:37:10.572702 12928 raft_consensus.cc:374] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:10.572793 12928 raft_consensus.cc:729] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Initialized, Role: FOLLOWER
I20260926 16:37:10.573321 12928 consensus_queue.cc:260] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.574873 12928 ts_tablet_manager.cc:1428] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260926 16:37:10.682909 12908 tablet.cc:2369] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:37:10.688951 12769 tablet.cc:2369] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:37:10.714437 12630 tablet.cc:2369] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:10.745994 12934 raft_consensus.cc:482] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:10.746318 12934 raft_consensus.cc:504] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.747944 12934 leader_election.cc:288] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553), b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:10.754474 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.748269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:55891 (local address 127.10.110.3:43553)
0926 16:37:10.748458 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.748464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.749209 (+ 745us) server_negotiation.cc:407] Connection header received
0926 16:37:10.749378 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.749384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.749457 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.749521 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:10.750103 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.750816 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.752773 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.753571 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.753709 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.753783 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.754118 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.754240 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:37:10.754487 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.748424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:47615 (local address 127.10.110.2:37263)
0926 16:37:10.748640 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.748646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.749287 (+ 641us) server_negotiation.cc:407] Connection header received
0926 16:37:10.749432 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.749436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.749490 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.749569 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:10.750197 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.750948 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.752816 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.753594 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.753749 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.753822 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.754137 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.754252 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:37:10.754652 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.748304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:43553 (local address 127.10.110.1:55891)
0926 16:37:10.749086 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.749109 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.749236 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.749555 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.749559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.749588 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:10.749958 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.749967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.750948 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.750954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.752631 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.752646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.753801 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.753807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.753938 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.753993 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.754064 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.754193 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":25,"thread_start_us":425,"threads_started":1}
I20260926 16:37:10.754896 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.748379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37263 (local address 127.10.110.1:47615)
0926 16:37:10.749177 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.749193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.749320 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.749597 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.749602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.749624 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:10.750011 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.750021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.751124 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.751131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.752669 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.752681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.753850 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.753857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.753988 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.754010 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.754100 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.754222 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":9,"thread_start_us":474,"threads_started":1}
I20260926 16:37:10.755612 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85010dbcea8440409d9696e05e477083" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" is_pre_election: true
I20260926 16:37:10.755798 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85010dbcea8440409d9696e05e477083" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb" is_pre_election: true
I20260926 16:37:10.756260 12857 raft_consensus.cc:2454] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 0.
I20260926 16:37:10.756268 12718 raft_consensus.cc:2454] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 0.
I20260926 16:37:10.756893 12512 leader_election.cc:302] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, b362bac792784b3ca2511e75dcdb07eb; no voters:
I20260926 16:37:10.757272 12934 raft_consensus.cc:2781] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:10.757395 12934 raft_consensus.cc:482] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:10.757488 12934 raft_consensus.cc:3037] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:10.759593 12934 raft_consensus.cc:504] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.760169 12934 leader_election.cc:288] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 election: Requested vote from peers e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553), b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:10.760560 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85010dbcea8440409d9696e05e477083" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb"
I20260926 16:37:10.760567 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85010dbcea8440409d9696e05e477083" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb"
I20260926 16:37:10.760746 12857 raft_consensus.cc:3037] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:10.760747 12718 raft_consensus.cc:3037] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:10.762782 12718 raft_consensus.cc:2454] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 1.
I20260926 16:37:10.762821 12857 raft_consensus.cc:2454] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 1.
I20260926 16:37:10.763149 12512 leader_election.cc:302] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, b362bac792784b3ca2511e75dcdb07eb; no voters:
I20260926 16:37:10.763402 12934 raft_consensus.cc:2781] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:10.763778 12934 raft_consensus.cc:686] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 1 LEADER]: Becoming Leader. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Running, Role: LEADER
I20260926 16:37:10.764235 12934 consensus_queue.cc:237] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } }
I20260926 16:37:10.768364 12420 catalog_manager.cc:5495] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1). New cstate: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:37:10 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440628, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:10.838176 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.826145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32813 (local address 127.0.0.1:34724)
0926 16:37:10.826696 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:10.826711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:10.826828 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:10.827083 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:10.827086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:10.827800 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:10.828074 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.828083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.828997 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.829001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.829962 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:10.829973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.830678 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.830683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:10.830786 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.831350 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:10.831366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:10.833432 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:10.836023 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:10.836033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:10.836038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:10.836409 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:10.836713 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:10.836717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:10.836719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:10.836854 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:10.837323 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:10.837329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:10.837474 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 16:37:10.837629 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 16:37:10.837833 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":301,"threads_started":1}
I20260926 16:37:10.838195 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:10.826141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.10.110.62:32813)
0926 16:37:10.826330 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:10.826335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:10.826789 (+ 454us) server_negotiation.cc:407] Connection header received
0926 16:37:10.826914 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:10.826918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:10.826976 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:10.827052 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:10.828208 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.828891 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.830081 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:10.830513 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:10.830599 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:10.833587 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:10.833592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:10.833595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:10.833622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:10.835902 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.836499 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.836502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.836505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.836620 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:10.836943 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:10.836946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:10.836948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:10.837211 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:10.837312 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:10.837733 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:37:10.837957 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":68}
I20260926 16:37:10.843027 12420 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:34724:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:37:10.844101 12420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:10.850289 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 73784e70667f491cb3937e14f38b92d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=030a744c3b6243d78320b2239ecfedbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:10.850210 12559 tablet_service.cc:1467] Processing CreateTablet for tablet 73784e70667f491cb3937e14f38b92d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=030a744c3b6243d78320b2239ecfedbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:10.850481 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 73784e70667f491cb3937e14f38b92d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=030a744c3b6243d78320b2239ecfedbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:10.851161 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73784e70667f491cb3937e14f38b92d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:10.851162 12559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73784e70667f491cb3937e14f38b92d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:10.851169 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73784e70667f491cb3937e14f38b92d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:10.856541 12926 tablet_bootstrap.cc:492] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: Bootstrap starting.
I20260926 16:37:10.857856 12926 tablet_bootstrap.cc:654] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:10.857919 12927 tablet_bootstrap.cc:492] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: Bootstrap starting.
I20260926 16:37:10.858290 12928 tablet_bootstrap.cc:492] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: Bootstrap starting.
I20260926 16:37:10.859416 12927 tablet_bootstrap.cc:654] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:10.859598 12928 tablet_bootstrap.cc:654] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:10.859890 12926 tablet_bootstrap.cc:492] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: No bootstrap required, opened a new log
I20260926 16:37:10.860025 12926 ts_tablet_manager.cc:1397] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260926 16:37:10.860677 12926 raft_consensus.cc:348] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:10.860863 12926 raft_consensus.cc:374] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:10.860962 12926 raft_consensus.cc:729] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Initialized, Role: FOLLOWER
I20260926 16:37:10.861222 12926 consensus_queue.cc:260] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:10.861598 12927 tablet_bootstrap.cc:492] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: No bootstrap required, opened a new log
I20260926 16:37:10.861758 12927 ts_tablet_manager.cc:1397] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 16:37:10.861829 12926 ts_tablet_manager.cc:1428] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:37:10.861733 12932 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:10.862136 12932 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:10.862478 12927 raft_consensus.cc:348] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:10.862748 12927 raft_consensus.cc:374] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:10.862844 12927 raft_consensus.cc:729] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b362bac792784b3ca2511e75dcdb07eb, State: Initialized, Role: FOLLOWER
I20260926 16:37:10.863113 12927 consensus_queue.cc:260] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:10.863524 12933 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:10.863811 12928 tablet_bootstrap.cc:492] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: No bootstrap required, opened a new log
I20260926 16:37:10.863896 12933 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:10.863984 12928 ts_tablet_manager.cc:1397] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260926 16:37:10.864475 12927 ts_tablet_manager.cc:1428] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 16:37:10.864861 12928 raft_consensus.cc:348] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:10.865097 12928 raft_consensus.cc:374] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:10.865193 12928 raft_consensus.cc:729] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Initialized, Role: FOLLOWER
I20260926 16:37:10.865438 12928 consensus_queue.cc:260] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:10.865836 12934 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:10.866004 12928 ts_tablet_manager.cc:1428] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 16:37:10.866142 12934 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:11.007249 12934 raft_consensus.cc:482] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:11.007455 12934 raft_consensus.cc:504] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.008198 12934 leader_election.cc:288] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263), e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553)
I20260926 16:37:11.008689 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73784e70667f491cb3937e14f38b92d5" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb" is_pre_election: true
I20260926 16:37:11.008738 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73784e70667f491cb3937e14f38b92d5" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" is_pre_election: true
I20260926 16:37:11.008917 12718 raft_consensus.cc:2454] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 0.
I20260926 16:37:11.008946 12857 raft_consensus.cc:2454] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 0.
I20260926 16:37:11.009410 12512 leader_election.cc:302] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, b362bac792784b3ca2511e75dcdb07eb, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:11.009754 12934 raft_consensus.cc:2781] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:11.009867 12934 raft_consensus.cc:482] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:11.009987 12934 raft_consensus.cc:3037] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:11.012945 12934 raft_consensus.cc:504] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.013557 12934 leader_election.cc:288] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 election: Requested vote from peers b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263), e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553)
I20260926 16:37:11.013984 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73784e70667f491cb3937e14f38b92d5" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb"
I20260926 16:37:11.014070 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73784e70667f491cb3937e14f38b92d5" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb"
I20260926 16:37:11.014225 12718 raft_consensus.cc:3037] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:11.014274 12857 raft_consensus.cc:3037] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:11.017047 12718 raft_consensus.cc:2454] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 1.
I20260926 16:37:11.017047 12857 raft_consensus.cc:2454] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 1.
I20260926 16:37:11.017501 12509 leader_election.cc:302] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:11.017769 12934 raft_consensus.cc:2781] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:11.017946 12934 raft_consensus.cc:686] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [term 1 LEADER]: Becoming Leader. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Running, Role: LEADER
I20260926 16:37:11.018177 12934 consensus_queue.cc:237] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.019794 12937 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: TxnStatusTablet state changed. Reason: New leader 3a5e78fd3f084adc96b78a863b76234f. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.020219 12937 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:11.020852 12949 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:37:11.021106 12949 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:37:11.021302 12420 catalog_manager.cc:5495] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1). New cstate: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:11.063268 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.057438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34903 (local address 127.0.0.1:53430)
0926 16:37:11.057665 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:11.057686 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:11.057815 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:11.058668 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:11.058673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:11.058686 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:11.058854 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.058861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.059920 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.059925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.060981 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.061002 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.061700 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.061706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.061854 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.061906 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.062758 (+ 852us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.062778 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:37:11.062854 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:37:11.063004 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 16:37:11.063298 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.057646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.10.110.1:34903)
0926 16:37:11.058305 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.058309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.058323 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:37:11.058395 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.058399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.058483 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.058628 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:11.058994 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.059794 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.061141 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.061563 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.061665 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.062084 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.062659 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.062780 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.062885 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.062960 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":417,"threads_started":1}
I20260926 16:37:11.070748 12718 raft_consensus.cc:1261] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Refusing update from remote peer 3a5e78fd3f084adc96b78a863b76234f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:11.070914 12857 raft_consensus.cc:1261] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Refusing update from remote peer 3a5e78fd3f084adc96b78a863b76234f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:11.071699 12937 consensus_queue.cc:1035] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:11.072079 12934 consensus_queue.cc:1035] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:11.081074 12933 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: TxnStatusTablet state changed. Reason: New leader 3a5e78fd3f084adc96b78a863b76234f. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.082226 12933 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:11.084434 12934 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.084733 12934 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:11.091445 12933 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.091682 12933 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P b362bac792784b3ca2511e75dcdb07eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:11.100656 12932 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: TxnStatusTablet state changed. Reason: New leader 3a5e78fd3f084adc96b78a863b76234f. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.100930 12932 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:11.102456 12937 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.102725 12937 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P 3a5e78fd3f084adc96b78a863b76234f: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:11.105909 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.098475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.10.110.1:34903)
0926 16:37:11.098640 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.098647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.099208 (+ 561us) server_negotiation.cc:407] Connection header received
0926 16:37:11.099386 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.099392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.099483 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.099575 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:11.100766 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.101834 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.103859 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.104475 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.104570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.105172 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.105323 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.105421 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.105514 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.105601 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:37:11.117616 12932 tablet_replica.cc:439] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } } }
I20260926 16:37:11.117933 12932 tablet_replica.cc:442] T 73784e70667f491cb3937e14f38b92d5 P e8e4c6c3b79a4eb4880f44415ef934fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:11.129859 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.115687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:40343 (local address 127.10.110.1:34903)
0926 16:37:11.115879 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.115885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.124448 (+ 8563us) server_negotiation.cc:407] Connection header received
0926 16:37:11.124636 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.124641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.124750 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.124853 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:11.125338 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.126487 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.127932 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.128581 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.128705 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.129098 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.129222 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.129311 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.129412 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.129511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:37:11.129880 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.115576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34903 (local address 127.10.110.1:40343)
0926 16:37:11.124346 (+ 8770us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:11.124363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:11.124495 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:11.124861 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:11.124864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:11.124873 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:11.125191 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.125200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.126638 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.126642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.127793 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.127804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.128744 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.128748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.128859 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.128917 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.129311 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.129318 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:37:11.129378 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 16:37:11.129479 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8596,"thread_start_us":8467,"threads_started":1}
I20260926 16:37:11.139175 12856 raft_consensus.cc:1261] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Refusing update from remote peer 3a5e78fd3f084adc96b78a863b76234f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:11.139289 12718 raft_consensus.cc:1261] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Refusing update from remote peer 3a5e78fd3f084adc96b78a863b76234f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:11.139883 12937 consensus_queue.cc:1035] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:11.140336 12934 consensus_queue.cc:1035] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:11.144860 12953 mvcc.cc:205] Tried to move back new op lower bound from 7333644825139740672 to 7333644823613566976. Current Snapshot: MvccSnapshot[applied={T|T < 7333644825139740672}]
I20260926 16:37:11.195997 12420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:11.196242 12420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:11.199199 12420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Sending DeleteTablet for 3 replicas of tablet 85010dbcea8440409d9696e05e477083
I20260926 16:37:11.200321 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 85010dbcea8440409d9696e05e477083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:11 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260926 16:37:11.200348 12837 tablet_service.cc:1514] Processing DeleteTablet for tablet 85010dbcea8440409d9696e05e477083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:11 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:44308
I20260926 16:37:11.201215 12971 tablet_replica.cc:330] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: stopping tablet replica
I20260926 16:37:11.201234 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet 85010dbcea8440409d9696e05e477083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:11 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 16:37:11.201591 12971 raft_consensus.cc:2229] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:11.202018 12971 raft_consensus.cc:2258] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:11.202260 12973 tablet_replica.cc:330] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: stopping tablet replica
I20260926 16:37:11.202652 12973 raft_consensus.cc:2229] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:11.203326 12973 raft_consensus.cc:2258] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:11.203490 12420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:11.204200 12420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:11.205255 12973 ts_tablet_manager.cc:1905] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:11.211915 12836 tablet_service.cc:1467] Processing CreateTablet for tablet d0cdf1c93d59409f884be8b276bc1534 (DEFAULT_TABLE table=test-table [id=acddf9a4eb1242e3a2224b4f9e9f51d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:11.212417 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cdf1c93d59409f884be8b276bc1534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:11.212731 12973 ts_tablet_manager.cc:1918] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:11.212891 12973 log.cc:1196] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/85010dbcea8440409d9696e05e477083
I20260926 16:37:11.213378 12973 ts_tablet_manager.cc:1939] T 85010dbcea8440409d9696e05e477083 P 3a5e78fd3f084adc96b78a863b76234f: Deleting consensus metadata
I20260926 16:37:11.216383 12407 catalog_manager.cc:4841] TS 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903): tablet 85010dbcea8440409d9696e05e477083 (table test-table [id=164a9fe8f1f648d696c5711a17565fc5]) successfully deleted
I20260926 16:37:11.216776 12558 tablet_service.cc:1467] Processing CreateTablet for tablet d0cdf1c93d59409f884be8b276bc1534 (DEFAULT_TABLE table=test-table [id=acddf9a4eb1242e3a2224b4f9e9f51d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:11.216899 12972 tablet_replica.cc:330] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: stopping tablet replica
I20260926 16:37:11.217209 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cdf1c93d59409f884be8b276bc1534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:11.217203 12698 tablet_service.cc:1467] Processing CreateTablet for tablet d0cdf1c93d59409f884be8b276bc1534 (DEFAULT_TABLE table=test-table [id=acddf9a4eb1242e3a2224b4f9e9f51d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:11.217577 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0cdf1c93d59409f884be8b276bc1534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:11.218338 12972 raft_consensus.cc:2229] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:11.218693 12972 raft_consensus.cc:2258] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:11.221544 12972 ts_tablet_manager.cc:1905] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:11.223387 12928 tablet_bootstrap.cc:492] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Bootstrap starting.
I20260926 16:37:11.225050 12928 tablet_bootstrap.cc:654] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:11.226696 12927 tablet_bootstrap.cc:492] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Bootstrap starting.
I20260926 16:37:11.228353 12927 tablet_bootstrap.cc:654] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:11.231091 12972 ts_tablet_manager.cc:1918] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:11.231249 12972 log.cc:1196] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/85010dbcea8440409d9696e05e477083
I20260926 16:37:11.231765 12972 ts_tablet_manager.cc:1939] T 85010dbcea8440409d9696e05e477083 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting consensus metadata
I20260926 16:37:11.231840 12926 tablet_bootstrap.cc:492] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Bootstrap starting.
I20260926 16:37:11.233568 12926 tablet_bootstrap.cc:654] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:11.233830 12406 catalog_manager.cc:4841] TS e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553): tablet 85010dbcea8440409d9696e05e477083 (table test-table [id=164a9fe8f1f648d696c5711a17565fc5]) successfully deleted
I20260926 16:37:11.238701 12971 ts_tablet_manager.cc:1905] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:11.245157 12926 tablet_bootstrap.cc:492] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: No bootstrap required, opened a new log
I20260926 16:37:11.245381 12926 ts_tablet_manager.cc:1397] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260926 16:37:11.246044 12927 tablet_bootstrap.cc:492] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: No bootstrap required, opened a new log
I20260926 16:37:11.246249 12927 ts_tablet_manager.cc:1397] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20260926 16:37:11.246379 12926 raft_consensus.cc:348] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.246704 12926 raft_consensus.cc:374] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:11.246803 12926 raft_consensus.cc:729] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Initialized, Role: FOLLOWER
I20260926 16:37:11.247113 12926 consensus_queue.cc:260] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.247125 12927 raft_consensus.cc:348] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.247419 12927 raft_consensus.cc:374] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:11.247555 12927 raft_consensus.cc:729] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b362bac792784b3ca2511e75dcdb07eb, State: Initialized, Role: FOLLOWER
I20260926 16:37:11.247879 12927 consensus_queue.cc:260] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.249322 12927 ts_tablet_manager.cc:1428] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 16:37:11.250286 12926 ts_tablet_manager.cc:1428] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260926 16:37:11.255695 12971 ts_tablet_manager.cc:1918] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:11.255964 12971 log.cc:1196] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/85010dbcea8440409d9696e05e477083
I20260926 16:37:11.256359 12928 tablet_bootstrap.cc:492] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: No bootstrap required, opened a new log
I20260926 16:37:11.256415 12971 ts_tablet_manager.cc:1939] T 85010dbcea8440409d9696e05e477083 P b362bac792784b3ca2511e75dcdb07eb: Deleting consensus metadata
I20260926 16:37:11.256517 12928 ts_tablet_manager.cc:1397] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.017s
I20260926 16:37:11.257143 12928 raft_consensus.cc:348] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.257346 12928 raft_consensus.cc:374] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:11.257427 12928 raft_consensus.cc:729] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Initialized, Role: FOLLOWER
I20260926 16:37:11.257673 12928 consensus_queue.cc:260] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.258090 12409 catalog_manager.cc:4841] TS b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263): tablet 85010dbcea8440409d9696e05e477083 (table test-table [id=164a9fe8f1f648d696c5711a17565fc5]) successfully deleted
I20260926 16:37:11.258154 12928 ts_tablet_manager.cc:1428] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 16:37:11.481689 12932 raft_consensus.cc:482] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:11.481892 12932 raft_consensus.cc:504] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.483481 12932 leader_election.cc:288] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903), b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:11.504360 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.484044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:48705 (local address 127.10.110.1:34903)
0926 16:37:11.484253 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.484260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.496311 (+ 12051us) server_negotiation.cc:407] Connection header received
0926 16:37:11.496776 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.496783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.496860 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.496925 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:11.497860 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.498994 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.501955 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.503361 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.503560 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.503639 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.503938 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.504077 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:37:11.504472 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.483693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34903 (local address 127.10.110.3:48705)
0926 16:37:11.496156 (+ 12463us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:11.496184 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:11.496602 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:11.497118 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:11.497123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:11.497166 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:11.497657 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.497675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.499190 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.499195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.501718 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.501735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.503558 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.503564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.503736 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.503785 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 16:37:11.503897 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 16:37:11.504119 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12320,"mutex_wait_us":208,"spinlock_wait_cycles":1408,"thread_start_us":12182,"threads_started":1}
I20260926 16:37:11.505311 12579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cdf1c93d59409f884be8b276bc1534" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5e78fd3f084adc96b78a863b76234f" is_pre_election: true
I20260926 16:37:11.505733 12579 raft_consensus.cc:2454] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 0.
I20260926 16:37:11.506165 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.484426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:40109 (local address 127.10.110.2:37263)
0926 16:37:11.489909 (+ 5483us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.489916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.496594 (+ 6678us) server_negotiation.cc:407] Connection header received
0926 16:37:11.496799 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.496803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.496869 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.496929 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:11.497911 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.499001 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.503730 (+ 4729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.504955 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.505122 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.505194 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.505729 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.505874 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5356,"thread_start_us":5218,"threads_started":1}
I20260926 16:37:11.506222 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.484111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37263 (local address 127.10.110.3:40109)
0926 16:37:11.496486 (+ 12375us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:11.496503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:11.496668 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:11.497118 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:11.497123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:11.497175 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:11.497736 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.497747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.499191 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.499195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.501749 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.501758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.505321 (+ 3563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.505328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.505514 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.505559 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 16:37:11.505689 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 16:37:11.505823 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12245,"mutex_wait_us":198,"spinlock_wait_cycles":5760,"thread_start_us":12212,"threads_started":1}
I20260926 16:37:11.506613 12789 leader_election.cc:302] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:11.507015 12932 raft_consensus.cc:2781] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:11.507017 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cdf1c93d59409f884be8b276bc1534" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb" is_pre_election: true
I20260926 16:37:11.507139 12932 raft_consensus.cc:482] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:11.507220 12932 raft_consensus.cc:3037] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:11.507233 12718 raft_consensus.cc:2454] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 0.
I20260926 16:37:11.509531 12932 raft_consensus.cc:504] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.510088 12932 leader_election.cc:288] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 election: Requested vote from peers 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903), b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:11.510547 12579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cdf1c93d59409f884be8b276bc1534" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5e78fd3f084adc96b78a863b76234f"
I20260926 16:37:11.510619 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0cdf1c93d59409f884be8b276bc1534" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb"
I20260926 16:37:11.510789 12718 raft_consensus.cc:3037] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:11.510782 12579 raft_consensus.cc:3037] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:11.513684 12718 raft_consensus.cc:2454] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 1.
I20260926 16:37:11.513727 12579 raft_consensus.cc:2454] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 1.
I20260926 16:37:11.514091 12791 leader_election.cc:302] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b362bac792784b3ca2511e75dcdb07eb, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:11.514390 12932 raft_consensus.cc:2781] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:11.514874 12932 raft_consensus.cc:686] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 LEADER]: Becoming Leader. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Running, Role: LEADER
I20260926 16:37:11.515422 12932 consensus_queue.cc:237] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:11.519111 12420 catalog_manager.cc:5495] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3). New cstate: current_term: 1 leader_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } health_report { overall_health: HEALTHY } } }
I20260926 16:37:11.597735 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.591924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.10.110.3:43553)
0926 16:37:11.592770 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.592775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.592795 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:37:11.592876 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.592880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.592950 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.593102 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:11.593523 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.594293 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.595597 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.596025 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.596132 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.596552 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.597109 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.597238 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.597340 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.597422 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":497,"threads_started":1}
I20260926 16:37:11.613260 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.607433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:59015 (local address 127.10.110.1:34903)
0926 16:37:11.607574 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.607587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.607956 (+ 369us) server_negotiation.cc:407] Connection header received
0926 16:37:11.608142 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.608145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.608210 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.608286 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:11.608801 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.609518 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.611508 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.612048 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.612144 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.612643 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.612770 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.612834 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.612959 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.613025 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:37:11.613471 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.607286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34903 (local address 127.10.110.3:59015)
0926 16:37:11.607872 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:11.607892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:11.608028 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:11.608314 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:11.608318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:11.608330 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:11.608668 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.608679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.609644 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.609649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.611377 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.611392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.612220 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.612227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.612407 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.612475 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.612858 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.612869 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:37:11.612946 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:37:11.613082 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":305,"threads_started":1}
I20260926 16:37:11.622037 12718 raft_consensus.cc:1261] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Refusing update from remote peer e8e4c6c3b79a4eb4880f44415ef934fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:11.622310 12579 raft_consensus.cc:1261] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Refusing update from remote peer e8e4c6c3b79a4eb4880f44415ef934fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:11.622640 12980 consensus_queue.cc:1035] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:11.622978 12932 consensus_queue.cc:1035] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:11.633354 12955 mvcc.cc:205] Tried to move back new op lower bound from 7333644827118288896 to 7333644826691309568. Current Snapshot: MvccSnapshot[applied={T|T < 7333644827118288896}]
I20260926 16:37:11.665426 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.656869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:44609 (local address 127.10.110.3:43553)
0926 16:37:11.657027 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:11.657033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:11.660591 (+ 3558us) server_negotiation.cc:407] Connection header received
0926 16:37:11.660667 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:11.660671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:11.660754 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:11.660840 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:11.661526 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.662329 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.663785 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.664243 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.664350 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.664754 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.664855 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.664923 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:11.665053 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:37:11.665134 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:37:11.665534 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:11.656870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:43553 (local address 127.10.110.1:44609)
0926 16:37:11.660319 (+ 3449us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:11.660340 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:11.660475 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:11.660877 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:11.660882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:11.660895 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:11.661346 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.661359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.662464 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.662469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.663649 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:11.663662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:11.664419 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:11.664430 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:11.664565 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:11.664611 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:11.664941 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:11.664952 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:37:11.665034 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:37:11.665173 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3276,"thread_start_us":3135,"threads_started":1}
W20260926 16:37:11.666759 12955 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:37:13.263943 12420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:13.264159 12420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:13.267948 12420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Sending DeleteTablet for 3 replicas of tablet d0cdf1c93d59409f884be8b276bc1534
I20260926 16:37:13.268807 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cdf1c93d59409f884be8b276bc1534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:13 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 16:37:13.268801 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cdf1c93d59409f884be8b276bc1534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:13 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260926 16:37:13.268986 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet d0cdf1c93d59409f884be8b276bc1534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:13 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:44308
I20260926 16:37:13.279532 13016 tablet_replica.cc:330] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: stopping tablet replica
I20260926 16:37:13.279742 13016 raft_consensus.cc:2229] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:13.280462 13016 raft_consensus.cc:2258] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:13.281292 13015 tablet_replica.cc:330] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: stopping tablet replica
I20260926 16:37:13.281481 13015 raft_consensus.cc:2229] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:13.281708 13015 raft_consensus.cc:2258] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:13.281893 13016 ts_tablet_manager.cc:1905] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:13.282251 13014 tablet_replica.cc:330] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: stopping tablet replica
I20260926 16:37:13.282419 13014 raft_consensus.cc:2229] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:13.282634 13014 raft_consensus.cc:2258] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:13.283324 13015 ts_tablet_manager.cc:1905] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:13.286036 13014 ts_tablet_manager.cc:1905] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:13.287269 13016 ts_tablet_manager.cc:1918] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:37:13.287454 13016 log.cc:1196] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/d0cdf1c93d59409f884be8b276bc1534
I20260926 16:37:13.287930 13016 ts_tablet_manager.cc:1939] T d0cdf1c93d59409f884be8b276bc1534 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting consensus metadata
I20260926 16:37:13.289112 12406 catalog_manager.cc:4841] TS e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553): tablet d0cdf1c93d59409f884be8b276bc1534 (table test-table [id=acddf9a4eb1242e3a2224b4f9e9f51d0]) successfully deleted
I20260926 16:37:13.289595 13015 ts_tablet_manager.cc:1918] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:37:13.289752 13015 log.cc:1196] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/d0cdf1c93d59409f884be8b276bc1534
I20260926 16:37:13.290262 13015 ts_tablet_manager.cc:1939] T d0cdf1c93d59409f884be8b276bc1534 P 3a5e78fd3f084adc96b78a863b76234f: Deleting consensus metadata
I20260926 16:37:13.291309 12407 catalog_manager.cc:4841] TS 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903): tablet d0cdf1c93d59409f884be8b276bc1534 (table test-table [id=acddf9a4eb1242e3a2224b4f9e9f51d0]) successfully deleted
I20260926 16:37:13.291651 13014 ts_tablet_manager.cc:1918] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:37:13.291769 13014 log.cc:1196] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/d0cdf1c93d59409f884be8b276bc1534
I20260926 16:37:13.292129 13014 ts_tablet_manager.cc:1939] T d0cdf1c93d59409f884be8b276bc1534 P b362bac792784b3ca2511e75dcdb07eb: Deleting consensus metadata
I20260926 16:37:13.292965 12409 catalog_manager.cc:4841] TS b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263): tablet d0cdf1c93d59409f884be8b276bc1534 (table test-table [id=acddf9a4eb1242e3a2224b4f9e9f51d0]) successfully deleted
Sep 26 16:37:13 dist-test-slave-whjp krb5kdc[12377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:37:13 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:13.325390 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:13.307961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.10.110.62:32813)
0926 16:37:13.308449 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:13.308454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:13.308506 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:37:13.308654 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:13.308659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:13.308728 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:13.308822 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:13.310103 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:13.310899 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:13.312215 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:13.312613 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:13.312712 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:13.320525 (+ 7813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:13.320530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:13.320534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:13.320573 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:13.323094 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:13.323699 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:13.323702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:13.323705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:13.323813 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:13.324162 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:13.324174 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:13.324176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:13.324433 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:13.324612 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:13.324930 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 16:37:13.325092 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
I20260926 16:37:13.328747 12420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:13.329339 12420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:13.335595 12558 tablet_service.cc:1467] Processing CreateTablet for tablet c4b7569e965c44f1aee9a3358ca0f0b8 (DEFAULT_TABLE table=test-table [id=cc38343f1d6f49c8ae762052da7f7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:13.335912 12836 tablet_service.cc:1467] Processing CreateTablet for tablet c4b7569e965c44f1aee9a3358ca0f0b8 (DEFAULT_TABLE table=test-table [id=cc38343f1d6f49c8ae762052da7f7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:13.335968 12698 tablet_service.cc:1467] Processing CreateTablet for tablet c4b7569e965c44f1aee9a3358ca0f0b8 (DEFAULT_TABLE table=test-table [id=cc38343f1d6f49c8ae762052da7f7e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:13.336122 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b7569e965c44f1aee9a3358ca0f0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:13.336292 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b7569e965c44f1aee9a3358ca0f0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:13.336305 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b7569e965c44f1aee9a3358ca0f0b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:13.341413 13025 tablet_bootstrap.cc:492] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Bootstrap starting.
I20260926 16:37:13.342929 13025 tablet_bootstrap.cc:654] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:13.343537 13026 tablet_bootstrap.cc:492] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Bootstrap starting.
I20260926 16:37:13.344411 13027 tablet_bootstrap.cc:492] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Bootstrap starting.
I20260926 16:37:13.345497 13025 tablet_bootstrap.cc:492] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: No bootstrap required, opened a new log
I20260926 16:37:13.345551 13026 tablet_bootstrap.cc:654] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:13.345636 13025 ts_tablet_manager.cc:1397] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 16:37:13.346010 13027 tablet_bootstrap.cc:654] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:13.346552 13025 raft_consensus.cc:348] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.346913 13025 raft_consensus.cc:374] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:13.347029 13025 raft_consensus.cc:729] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b362bac792784b3ca2511e75dcdb07eb, State: Initialized, Role: FOLLOWER
I20260926 16:37:13.347450 13026 tablet_bootstrap.cc:492] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: No bootstrap required, opened a new log
I20260926 16:37:13.347335 13025 consensus_queue.cc:260] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.347582 13026 ts_tablet_manager.cc:1397] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 16:37:13.348356 13026 raft_consensus.cc:348] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.348594 13026 raft_consensus.cc:374] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:13.348698 13026 raft_consensus.cc:729] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Initialized, Role: FOLLOWER
I20260926 16:37:13.348909 13026 consensus_queue.cc:260] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.349375 13026 ts_tablet_manager.cc:1428] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:37:13.349979 13025 ts_tablet_manager.cc:1428] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 16:37:13.352880 13027 tablet_bootstrap.cc:492] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: No bootstrap required, opened a new log
I20260926 16:37:13.353029 13027 ts_tablet_manager.cc:1397] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 16:37:13.353636 13027 raft_consensus.cc:348] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.353826 13027 raft_consensus.cc:374] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:13.353888 13027 raft_consensus.cc:729] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Initialized, Role: FOLLOWER
I20260926 16:37:13.354071 13027 consensus_queue.cc:260] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.354511 13027 ts_tablet_manager.cc:1428] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 16:37:13.357945 13009 raft_consensus.cc:482] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:13.358114 13009 raft_consensus.cc:504] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.358700 13009 leader_election.cc:288] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903), b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:13.359141 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b7569e965c44f1aee9a3358ca0f0b8" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb" is_pre_election: true
I20260926 16:37:13.359138 12579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b7569e965c44f1aee9a3358ca0f0b8" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5e78fd3f084adc96b78a863b76234f" is_pre_election: true
I20260926 16:37:13.359367 12718 raft_consensus.cc:2454] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 0.
I20260926 16:37:13.359364 12579 raft_consensus.cc:2454] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 0.
I20260926 16:37:13.359783 12791 leader_election.cc:302] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, b362bac792784b3ca2511e75dcdb07eb, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:13.360070 13009 raft_consensus.cc:2781] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:13.360175 13009 raft_consensus.cc:482] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:13.360263 13009 raft_consensus.cc:3037] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:13.362182 13009 raft_consensus.cc:504] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.362706 13009 leader_election.cc:288] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 election: Requested vote from peers 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903), b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263)
I20260926 16:37:13.363173 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b7569e965c44f1aee9a3358ca0f0b8" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb"
I20260926 16:37:13.363350 12718 raft_consensus.cc:3037] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:13.363363 12579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b7569e965c44f1aee9a3358ca0f0b8" candidate_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5e78fd3f084adc96b78a863b76234f"
I20260926 16:37:13.363581 12579 raft_consensus.cc:3037] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:13.365410 12718 raft_consensus.cc:2454] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 1.
I20260926 16:37:13.365684 12579 raft_consensus.cc:2454] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e4c6c3b79a4eb4880f44415ef934fb in term 1.
I20260926 16:37:13.365916 12791 leader_election.cc:302] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b362bac792784b3ca2511e75dcdb07eb, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:13.366233 13009 raft_consensus.cc:2781] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:13.366370 13009 raft_consensus.cc:686] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 LEADER]: Becoming Leader. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Running, Role: LEADER
I20260926 16:37:13.366643 13009 consensus_queue.cc:237] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.369131 12421 catalog_manager.cc:5495] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3). New cstate: current_term: 1 leader_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } health_report { overall_health: HEALTHY } } }
I20260926 16:37:13.397291 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:13.392180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.10.110.3:43553)
0926 16:37:13.392735 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:13.392739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:13.392756 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:37:13.392827 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:13.392830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:13.392886 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:13.392967 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:13.393353 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:13.394107 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:13.395464 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:13.396169 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:13.396269 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:13.396632 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:13.396746 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:13.396811 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:13.396892 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 16:37:13.396949 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":276,"threads_started":1}
I20260926 16:37:13.405424 12718 raft_consensus.cc:1261] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Refusing update from remote peer e8e4c6c3b79a4eb4880f44415ef934fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:13.405643 12579 raft_consensus.cc:1261] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Refusing update from remote peer e8e4c6c3b79a4eb4880f44415ef934fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:13.405896 13009 consensus_queue.cc:1035] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:13.406131 13005 consensus_queue.cc:1035] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:13.411278 12997 mvcc.cc:205] Tried to move back new op lower bound from 7333644834424414208 to 7333644834271997952. Current Snapshot: MvccSnapshot[applied={T|T < 7333644834424414208}]
I20260926 16:37:13.453015 12422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:13.453251 12422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:13.456022 12422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Sending DeleteTablet for 3 replicas of tablet c4b7569e965c44f1aee9a3358ca0f0b8
I20260926 16:37:13.456673 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b7569e965c44f1aee9a3358ca0f0b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:13 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 16:37:13.456838 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b7569e965c44f1aee9a3358ca0f0b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:13 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:44308
I20260926 16:37:13.456943 13015 tablet_replica.cc:330] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: stopping tablet replica
I20260926 16:37:13.456964 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet c4b7569e965c44f1aee9a3358ca0f0b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:13 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260926 16:37:13.457093 13016 tablet_replica.cc:330] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: stopping tablet replica
I20260926 16:37:13.457109 13015 raft_consensus.cc:2229] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:13.457196 13014 tablet_replica.cc:330] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: stopping tablet replica
I20260926 16:37:13.457273 13015 raft_consensus.cc:2258] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:13.457283 13016 raft_consensus.cc:2229] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:13.457357 13014 raft_consensus.cc:2229] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:13.457522 13014 raft_consensus.cc:2258] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:13.457559 13016 raft_consensus.cc:2258] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:13.458573 13015 ts_tablet_manager.cc:1905] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:13.458676 13016 ts_tablet_manager.cc:1905] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:13.458987 12422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:13.459582 12422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:13.465157 13015 ts_tablet_manager.cc:1918] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:13.465312 13015 log.cc:1196] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/c4b7569e965c44f1aee9a3358ca0f0b8
I20260926 16:37:13.465723 13015 ts_tablet_manager.cc:1939] T c4b7569e965c44f1aee9a3358ca0f0b8 P 3a5e78fd3f084adc96b78a863b76234f: Deleting consensus metadata
I20260926 16:37:13.466475 12698 tablet_service.cc:1467] Processing CreateTablet for tablet 4b78960fc9984b33bffea9633a9f77ec (DEFAULT_TABLE table=test-table [id=5066fdc5beae402286cb2f6d1361fdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:13.466895 12698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b78960fc9984b33bffea9633a9f77ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:13.467020 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 4b78960fc9984b33bffea9633a9f77ec (DEFAULT_TABLE table=test-table [id=5066fdc5beae402286cb2f6d1361fdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:13.467394 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b78960fc9984b33bffea9633a9f77ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:13.467567 13014 ts_tablet_manager.cc:1905] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:13.473475 13025 tablet_bootstrap.cc:492] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Bootstrap starting.
I20260926 16:37:13.473883 12407 catalog_manager.cc:4841] TS 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903): tablet c4b7569e965c44f1aee9a3358ca0f0b8 (table test-table [id=cc38343f1d6f49c8ae762052da7f7e7f]) successfully deleted
I20260926 16:37:13.474272 13014 ts_tablet_manager.cc:1918] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:13.474431 13014 log.cc:1196] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/c4b7569e965c44f1aee9a3358ca0f0b8
I20260926 16:37:13.474385 12558 tablet_service.cc:1467] Processing CreateTablet for tablet 4b78960fc9984b33bffea9633a9f77ec (DEFAULT_TABLE table=test-table [id=5066fdc5beae402286cb2f6d1361fdaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:13.474808 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b78960fc9984b33bffea9633a9f77ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:13.474834 13014 ts_tablet_manager.cc:1939] T c4b7569e965c44f1aee9a3358ca0f0b8 P b362bac792784b3ca2511e75dcdb07eb: Deleting consensus metadata
I20260926 16:37:13.474841 13025 tablet_bootstrap.cc:654] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:13.475873 12409 catalog_manager.cc:4841] TS b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263): tablet c4b7569e965c44f1aee9a3358ca0f0b8 (table test-table [id=cc38343f1d6f49c8ae762052da7f7e7f]) successfully deleted
I20260926 16:37:13.476387 13016 ts_tablet_manager.cc:1918] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:13.476521 13016 log.cc:1196] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/c4b7569e965c44f1aee9a3358ca0f0b8
I20260926 16:37:13.476917 13016 ts_tablet_manager.cc:1939] T c4b7569e965c44f1aee9a3358ca0f0b8 P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting consensus metadata
I20260926 16:37:13.481052 13027 tablet_bootstrap.cc:492] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Bootstrap starting.
I20260926 16:37:13.481401 13026 tablet_bootstrap.cc:492] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Bootstrap starting.
I20260926 16:37:13.482825 13027 tablet_bootstrap.cc:654] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:13.482833 13026 tablet_bootstrap.cc:654] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:13.484400 12406 catalog_manager.cc:4841] TS e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553): tablet c4b7569e965c44f1aee9a3358ca0f0b8 (table test-table [id=cc38343f1d6f49c8ae762052da7f7e7f]) successfully deleted
I20260926 16:37:13.485383 13025 tablet_bootstrap.cc:492] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: No bootstrap required, opened a new log
I20260926 16:37:13.485550 13025 ts_tablet_manager.cc:1397] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260926 16:37:13.486078 13026 tablet_bootstrap.cc:492] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: No bootstrap required, opened a new log
I20260926 16:37:13.486215 13026 ts_tablet_manager.cc:1397] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 16:37:13.486296 13025 raft_consensus.cc:348] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.486570 13025 raft_consensus.cc:374] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:13.486644 13025 raft_consensus.cc:729] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b362bac792784b3ca2511e75dcdb07eb, State: Initialized, Role: FOLLOWER
I20260926 16:37:13.486857 13026 raft_consensus.cc:348] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.486929 13025 consensus_queue.cc:260] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.487083 13026 raft_consensus.cc:374] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:13.487149 13026 raft_consensus.cc:729] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e4c6c3b79a4eb4880f44415ef934fb, State: Initialized, Role: FOLLOWER
I20260926 16:37:13.487430 13025 ts_tablet_manager.cc:1428] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 16:37:13.487337 13026 consensus_queue.cc:260] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.487923 13026 ts_tablet_manager.cc:1428] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 16:37:13.492149 13027 tablet_bootstrap.cc:492] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: No bootstrap required, opened a new log
I20260926 16:37:13.492321 13027 ts_tablet_manager.cc:1397] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260926 16:37:13.492946 13027 raft_consensus.cc:348] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.493165 13027 raft_consensus.cc:374] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:13.493254 13027 raft_consensus.cc:729] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Initialized, Role: FOLLOWER
I20260926 16:37:13.493458 13027 consensus_queue.cc:260] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.493912 13027 ts_tablet_manager.cc:1428] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 16:37:13.539283 12991 raft_consensus.cc:482] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:13.539484 12991 raft_consensus.cc:504] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.540124 12991 leader_election.cc:288] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263), e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553)
I20260926 16:37:13.540621 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b78960fc9984b33bffea9633a9f77ec" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" is_pre_election: true
I20260926 16:37:13.540639 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b78960fc9984b33bffea9633a9f77ec" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb" is_pre_election: true
I20260926 16:37:13.540863 12856 raft_consensus.cc:2454] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 0.
I20260926 16:37:13.540863 12718 raft_consensus.cc:2454] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 0.
I20260926 16:37:13.541236 12512 leader_election.cc:302] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, b362bac792784b3ca2511e75dcdb07eb; no voters:
I20260926 16:37:13.541476 12991 raft_consensus.cc:2781] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:13.541574 12991 raft_consensus.cc:482] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:13.541635 12991 raft_consensus.cc:3037] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:13.543587 12991 raft_consensus.cc:504] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.544121 12991 leader_election.cc:288] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 election: Requested vote from peers b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263), e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553)
I20260926 16:37:13.544608 12718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b78960fc9984b33bffea9633a9f77ec" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b362bac792784b3ca2511e75dcdb07eb"
I20260926 16:37:13.544615 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b78960fc9984b33bffea9633a9f77ec" candidate_uuid: "3a5e78fd3f084adc96b78a863b76234f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb"
I20260926 16:37:13.544800 12718 raft_consensus.cc:3037] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:13.544817 12856 raft_consensus.cc:3037] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:13.546694 12856 raft_consensus.cc:2454] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 1.
I20260926 16:37:13.546787 12718 raft_consensus.cc:2454] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5e78fd3f084adc96b78a863b76234f in term 1.
I20260926 16:37:13.547075 12509 leader_election.cc:302] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5e78fd3f084adc96b78a863b76234f, e8e4c6c3b79a4eb4880f44415ef934fb; no voters:
I20260926 16:37:13.547367 12991 raft_consensus.cc:2781] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:13.547487 12991 raft_consensus.cc:686] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 1 LEADER]: Becoming Leader. State: Replica: 3a5e78fd3f084adc96b78a863b76234f, State: Running, Role: LEADER
I20260926 16:37:13.547737 12991 consensus_queue.cc:237] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } }
I20260926 16:37:13.550202 12420 catalog_manager.cc:5495] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1). New cstate: current_term: 1 leader_uuid: "3a5e78fd3f084adc96b78a863b76234f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5e78fd3f084adc96b78a863b76234f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:13.575346 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:13.570610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.10.110.1:34903)
0926 16:37:13.571069 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:13.571073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:13.571089 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:37:13.571133 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:13.571136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:13.571209 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:13.571288 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:13.571659 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:13.572451 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:13.573728 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:13.574126 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:13.574240 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:13.574655 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:13.574753 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:13.574824 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:13.574977 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 16:37:13.575051 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":249,"threads_started":1}
I20260926 16:37:13.583211 12856 raft_consensus.cc:1261] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Refusing update from remote peer 3a5e78fd3f084adc96b78a863b76234f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:13.583221 12718 raft_consensus.cc:1261] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Refusing update from remote peer 3a5e78fd3f084adc96b78a863b76234f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:13.583743 13000 consensus_queue.cc:1035] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b362bac792784b3ca2511e75dcdb07eb" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:13.584095 12995 consensus_queue.cc:1035] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e4c6c3b79a4eb4880f44415ef934fb" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:37:13.601977 12993 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:37:15.174582 12422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:15.174811 12422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:15.178685 12422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37832cb179a3465084c91db58e56ecbf: Sending DeleteTablet for 3 replicas of tablet 4b78960fc9984b33bffea9633a9f77ec
I20260926 16:37:15.179383 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b78960fc9984b33bffea9633a9f77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:15 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53426
I20260926 16:37:15.179490 12698 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b78960fc9984b33bffea9633a9f77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:15 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:35428
I20260926 16:37:15.179559 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b78960fc9984b33bffea9633a9f77ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:15 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:44308
I20260926 16:37:15.180233 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12493
I20260926 16:37:15.191921 13066 tablet_replica.cc:330] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: stopping tablet replica
I20260926 16:37:15.192188 13067 tablet_replica.cc:330] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: stopping tablet replica
I20260926 16:37:15.192378 13067 raft_consensus.cc:2229] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:15.192430 13068 tablet_replica.cc:330] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: stopping tablet replica
I20260926 16:37:15.192620 13067 raft_consensus.cc:2258] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:15.192633 13068 raft_consensus.cc:2229] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:15.192867 13068 raft_consensus.cc:2258] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:15.193145 13066 raft_consensus.cc:2229] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:15.193455 13066 raft_consensus.cc:2258] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:15.193905 13067 ts_tablet_manager.cc:1905] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:15.194653 13068 ts_tablet_manager.cc:1905] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:15.195350 13066 ts_tablet_manager.cc:1905] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:15.200011 13067 ts_tablet_manager.cc:1918] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:37:15.200203 13067 log.cc:1196] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/4b78960fc9984b33bffea9633a9f77ec
I20260926 16:37:15.200701 13067 ts_tablet_manager.cc:1939] T 4b78960fc9984b33bffea9633a9f77ec P b362bac792784b3ca2511e75dcdb07eb: Deleting consensus metadata
I20260926 16:37:15.200701 13068 ts_tablet_manager.cc:1918] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:37:15.200817 13068 log.cc:1196] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/4b78960fc9984b33bffea9633a9f77ec
I20260926 16:37:15.201256 13068 ts_tablet_manager.cc:1939] T 4b78960fc9984b33bffea9633a9f77ec P e8e4c6c3b79a4eb4880f44415ef934fb: Deleting consensus metadata
I20260926 16:37:15.201737 13066 ts_tablet_manager.cc:1918] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:37:15.201905 13066 log.cc:1196] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/4b78960fc9984b33bffea9633a9f77ec
I20260926 16:37:15.201869 12409 catalog_manager.cc:4841] TS b362bac792784b3ca2511e75dcdb07eb (127.10.110.2:37263): tablet 4b78960fc9984b33bffea9633a9f77ec (table test-table [id=5066fdc5beae402286cb2f6d1361fdaf]) successfully deleted
I20260926 16:37:15.202333 13066 ts_tablet_manager.cc:1939] T 4b78960fc9984b33bffea9633a9f77ec P 3a5e78fd3f084adc96b78a863b76234f: Deleting consensus metadata
I20260926 16:37:15.202334 12406 catalog_manager.cc:4841] TS e8e4c6c3b79a4eb4880f44415ef934fb (127.10.110.3:43553): tablet 4b78960fc9984b33bffea9633a9f77ec (table test-table [id=5066fdc5beae402286cb2f6d1361fdaf]) successfully deleted
Sep 26 16:37:15 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:15.203315 12407 catalog_manager.cc:4841] TS 3a5e78fd3f084adc96b78a863b76234f (127.10.110.1:34903): tablet 4b78960fc9984b33bffea9633a9f77ec (table test-table [id=5066fdc5beae402286cb2f6d1361fdaf]) successfully deleted
I20260926 16:37:15.208748 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:15.181746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.10.110.1:34903)
0926 16:37:15.193176 (+ 11430us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:15.193182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:15.193208 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:37:15.193269 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:15.193274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:15.193350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:15.193478 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:15.195093 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:15.196242 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:15.198048 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:15.198683 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:15.203168 (+ 4485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:15.203683 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:15.203696 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:15.203712 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:15.203738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:15.206323 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:15.206933 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:15.206941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:15.206943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:15.207108 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:15.207426 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:15.207430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:15.207432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:15.207785 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:15.208025 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:15.208266 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 16:37:15.208425 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11286,"thread_start_us":11154,"threads_started":1}
W20260926 16:37:15.209373 12624 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57684
W20260926 16:37:15.210005 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:15.210216 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 12493
I20260926 16:37:15.230999 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12633
Sep 26 16:37:15 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:15.245327 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:15.231712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.10.110.2:37263)
0926 16:37:15.232245 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:15.232250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:15.232266 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:37:15.232315 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:15.232319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:15.232379 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:15.232470 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:15.233754 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:15.234482 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:15.235656 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:15.236168 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:15.236284 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:15.240157 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:15.240171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:15.240196 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:15.240229 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:15.242663 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:15.243277 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:15.243285 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:15.243287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:15.243415 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:15.243761 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:15.243765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:15.243767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:15.244227 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:15.244486 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:15.244781 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 16:37:15.244988 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":260,"threads_started":1}
W20260926 16:37:15.245965 12763 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44482
W20260926 16:37:15.246692 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12633: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:15.246820 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 12633
I20260926 16:37:15.260627 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12772
Sep 26 16:37:15 dist-test-slave-whjp krb5kdc[12377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440633, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:15.274689 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:15.261309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.10.110.3:43553)
0926 16:37:15.261843 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:15.261848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:15.261863 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:37:15.261913 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:15.261917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:15.261972 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:15.262060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:15.263271 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:15.264041 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:15.265260 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:15.265649 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:15.265764 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:15.269643 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:15.269658 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:15.269682 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:15.269719 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:15.272203 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:15.272783 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:15.272791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:15.272793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:15.272905 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:15.273245 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:15.273250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:15.273254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:15.273596 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:15.273883 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:15.274154 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:37:15.274365 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":244,"threads_started":1}
W20260926 16:37:15.275413 12902 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59002
W20260926 16:37:15.276223 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12772: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:15.276369 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 12772
I20260926 16:37:15.293359 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 12390
I20260926 16:37:15.294655 12462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:15.407940 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 12390
2026-09-26T16:37:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8098 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:15 dist-test-slave-whjp krb5kdc[13076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:15 dist-test-slave-whjp krb5kdc[13076](info): set up 2 sockets
Sep 26 16:37:15 dist-test-slave-whjp krb5kdc[13076](info): commencing operation
krb5kdc: starting...
W20260926 16:37:16.396661 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:16 dist-test-slave-whjp krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:16.565394 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:42313
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:42313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:16.870877 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:16.871177 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:16.871307 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:16.878352 13089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:16.878459 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:16.878537 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:16.878614 13089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:16.878669 13089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:16.888207 13089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:42313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:16.889472 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:16.890745 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:16.902133 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:16.902201 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:16.902207 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:16.902362 13089 server_base.cc:1034] running on GCE node
I20260926 16:37:16.903503 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:16.904915 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:16.906140 13089 hybrid_clock.cc:648] HybridClock initialized: now 1790440636906083 us; error 37 us; skew 500 ppm
I20260926 16:37:16.906503 13089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:16 dist-test-slave-whjp krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440636, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:16.909657 13089 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:16.911052 13089 webserver.cc:460] Webserver started at http://127.10.110.62:40173/ using document root <none> and password file <none>
I20260926 16:37:16.911717 13089 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:16.911847 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:16.912129 13089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:16.914961 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "ae45fb5f0e70436597ce433e27ba7db0"
format_stamp: "Formatted at 2026-09-26 16:37:16 on dist-test-slave-whjp"
server_key: "f989bf51712761fa504a9744c60811ab"
server_key_iv: "beafbbfa4443196e6bc287c8810f9dec"
server_key_version: "encryptionkey@0"
I20260926 16:37:16.915745 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "ae45fb5f0e70436597ce433e27ba7db0"
format_stamp: "Formatted at 2026-09-26 16:37:16 on dist-test-slave-whjp"
server_key: "f989bf51712761fa504a9744c60811ab"
server_key_iv: "beafbbfa4443196e6bc287c8810f9dec"
server_key_version: "encryptionkey@0"
I20260926 16:37:16.920856 13089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 16:37:16.924042 13104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:16.925292 13089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:37:16.925438 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "ae45fb5f0e70436597ce433e27ba7db0"
format_stamp: "Formatted at 2026-09-26 16:37:16 on dist-test-slave-whjp"
server_key: "f989bf51712761fa504a9744c60811ab"
server_key_iv: "beafbbfa4443196e6bc287c8810f9dec"
server_key_version: "encryptionkey@0"
I20260926 16:37:16.925599 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:16.945999 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:16.949699 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:16.949950 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:16.966719 13089 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:42313
I20260926 16:37:16.966744 13155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:42313 every 8 connection(s)
I20260926 16:37:16.968098 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:16.971019 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13089
I20260926 16:37:16.971215 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:16.971791 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:16.971804 10680 external_mini_cluster.cc:1428] Setting key d3a3957b5b0d4bd07a60bd6eec223b81
I20260926 16:37:16.985760 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0: Bootstrap starting.
Sep 26 16:37:16 dist-test-slave-whjp krb5kdc[13076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:16.989607 13156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:16.990967 13156 log.cc:824] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:16.993844 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0: No bootstrap required, opened a new log
I20260926 16:37:16.995014 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:16.973708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.10.110.62:42313)
0926 16:37:16.974484 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:16.974501 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:16.974558 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:37:16.976608 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:16.976645 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:16.977208 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:16.977678 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:16.978893 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:16.980849 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:16.982093 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:16.983204 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:16.983398 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:16.987175 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:16.987187 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:16.987210 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:16.987270 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:16.991004 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:16.991692 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:16.991700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:16.991706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:16.991928 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:16.992278 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:16.992282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:16.992285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:16.992862 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:16.993133 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:16.993542 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 16:37:16.993974 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":312,"threads_started":1}
I20260926 16:37:17.001653 13156 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae45fb5f0e70436597ce433e27ba7db0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42313 } }
I20260926 16:37:17.002000 13156 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:17.002092 13156 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae45fb5f0e70436597ce433e27ba7db0, State: Initialized, Role: FOLLOWER
I20260926 16:37:17.002616 13156 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae45fb5f0e70436597ce433e27ba7db0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42313 } }
I20260926 16:37:17.002830 13156 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:17.002955 13156 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:17.003099 13156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:17.005234 13156 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae45fb5f0e70436597ce433e27ba7db0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42313 } }
I20260926 16:37:17.005789 13156 leader_election.cc:302] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae45fb5f0e70436597ce433e27ba7db0; no voters:
I20260926 16:37:17.006353 13156 leader_election.cc:288] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:17.006430 13161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:17.007102 13161 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [term 1 LEADER]: Becoming Leader. State: Replica: ae45fb5f0e70436597ce433e27ba7db0, State: Running, Role: LEADER
I20260926 16:37:17.007598 13161 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae45fb5f0e70436597ce433e27ba7db0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42313 } }
I20260926 16:37:17.008388 13156 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:17.010095 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae45fb5f0e70436597ce433e27ba7db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae45fb5f0e70436597ce433e27ba7db0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42313 } } }
I20260926 16:37:17.010633 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:17.010455 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae45fb5f0e70436597ce433e27ba7db0. Latest consensus state: current_term: 1 leader_uuid: "ae45fb5f0e70436597ce433e27ba7db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae45fb5f0e70436597ce433e27ba7db0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42313 } } }
I20260926 16:37:17.010982 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:17.015360 13169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:17.019407 13169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:17.027292 13169 catalog_manager.cc:1347] Generated new cluster ID: 0bd16c6c5c1047f0b2ec4a51bcb505f4
I20260926 16:37:17.027392 13169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:17.045593 13169 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:17.046918 13169 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:17.052147 13169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae45fb5f0e70436597ce433e27ba7db0: Generated new TSK 0
I20260926 16:37:17.052899 13169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:17.091387 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:42313
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:17.399237 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:17.399499 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:17.399631 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:17.406769 13182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:17.406878 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:17.407038 13182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:17.417789 13182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:42313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:17.419091 13182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:17.420392 13182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:17.432775 13188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:17.432837 13190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:17.432811 13187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:17.433867 13182 server_base.cc:1034] running on GCE node
I20260926 16:37:17.434556 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:17.435580 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:17.436811 13182 hybrid_clock.cc:648] HybridClock initialized: now 1790440637436736 us; error 59 us; skew 500 ppm
I20260926 16:37:17.437124 13182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:17 dist-test-slave-whjp krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440637, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:17.440236 13182 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:17.441588 13182 webserver.cc:460] Webserver started at http://127.10.110.1:41355/ using document root <none> and password file <none>
I20260926 16:37:17.442240 13182 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:17.442348 13182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:17.442619 13182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:17.445454 13182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "04fc0ca85a78478380c2d0d647d2ba25"
format_stamp: "Formatted at 2026-09-26 16:37:17 on dist-test-slave-whjp"
server_key: "2d5985ebdcd1f9ff886706cd1226e9ca"
server_key_iv: "23436cb839b66b20191f41536912b6d4"
server_key_version: "encryptionkey@0"
I20260926 16:37:17.446182 13182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "04fc0ca85a78478380c2d0d647d2ba25"
format_stamp: "Formatted at 2026-09-26 16:37:17 on dist-test-slave-whjp"
server_key: "2d5985ebdcd1f9ff886706cd1226e9ca"
server_key_iv: "23436cb839b66b20191f41536912b6d4"
server_key_version: "encryptionkey@0"
I20260926 16:37:17.450896 13182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:37:17.454223 13197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:17.455523 13182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:17.455662 13182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "04fc0ca85a78478380c2d0d647d2ba25"
format_stamp: "Formatted at 2026-09-26 16:37:17 on dist-test-slave-whjp"
server_key: "2d5985ebdcd1f9ff886706cd1226e9ca"
server_key_iv: "23436cb839b66b20191f41536912b6d4"
server_key_version: "encryptionkey@0"
I20260926 16:37:17.455866 13182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:17.466811 13182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:17.470456 13182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:17.470705 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:17.471597 13182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:17.472908 13182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:17.472982 13182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:17.473094 13182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:17.473152 13182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:17.501127 13182 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:33401
I20260926 16:37:17.501163 13309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:33401 every 8 connection(s)
I20260926 16:37:17.502312 13182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:17.507508 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13182
I20260926 16:37:17.507696 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:17.508345 10680 external_mini_cluster.cc:1428] Setting key 0773afc1f6fbd3d5a24d2ce7380cc3e0
Sep 26 16:37:17 dist-test-slave-whjp krb5kdc[13076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440637, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:17.520823 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:17.504616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:35237 (local address 127.10.110.62:42313)
0926 16:37:17.505010 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:17.505015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:17.505850 (+ 835us) server_negotiation.cc:407] Connection header received
0926 16:37:17.507583 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:17.507588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:17.507665 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:17.507762 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:17.509388 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.510199 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.511674 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.512222 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.512340 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:17.515618 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:17.515623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:17.515626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:17.515680 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:17.518001 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:17.518620 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:17.518626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:17.518630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:17.518752 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:17.519112 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:17.519117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:17.519121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:17.519383 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:17.519504 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:17.520217 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 16:37:17.520445 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":199,"threads_started":1}
I20260926 16:37:17.521519 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:17.505047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:42313 (local address 127.10.110.1:35237)
0926 16:37:17.505702 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:17.505744 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:17.507337 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:17.508018 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:17.508025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:17.508584 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:17.509244 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:17.509266 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.510335 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.510341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:17.511539 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:17.511555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.512413 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.512421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:17.512667 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:17.513280 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:17.513301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:17.515416 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:17.518119 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:17.518128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:17.518143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:17.518516 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:17.518845 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:17.518850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:17.518854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:17.519015 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:17.519495 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:17.519502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:17.519804 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 16:37:17.520077 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 16:37:17.520485 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260926 16:37:17.522922 13310 heartbeater.cc:347] Connected to a master server at 127.10.110.62:42313
I20260926 16:37:17.523244 13310 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:37:17.524013 13310 heartbeater.cc:511] Master 127.10.110.62:42313 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:37:17.526008 13121 ts_manager.cc:194] Registered new tserver with Master: 04fc0ca85a78478380c2d0d647d2ba25 (127.10.110.1:33401)
I20260926 16:37:17.527437 13121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:35237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:17.544957 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:42313
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:17.853914 13316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:17.854162 13316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:17.854250 13316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:17.861320 13316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:17.861420 13316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:17.861536 13316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:17.870942 13316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:42313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:17.872149 13316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:17.873359 13316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:17.885133 13322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:17.885200 13321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:17.885169 13324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:17.886086 13316 server_base.cc:1034] running on GCE node
I20260926 16:37:17.886693 13316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:17.887607 13316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:17.888828 13316 hybrid_clock.cc:648] HybridClock initialized: now 1790440637888763 us; error 51 us; skew 500 ppm
I20260926 16:37:17.889181 13316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:17 dist-test-slave-whjp krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440637, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:17.892163 13316 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:17.893484 13316 webserver.cc:460] Webserver started at http://127.10.110.2:34375/ using document root <none> and password file <none>
I20260926 16:37:17.894241 13316 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:17.894353 13316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:17.894637 13316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:17.897418 13316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "f88c2a8749bf49a8b92d4ba87bdd0a48"
format_stamp: "Formatted at 2026-09-26 16:37:17 on dist-test-slave-whjp"
server_key: "d93fefbcdd90fb84153d079039aa869b"
server_key_iv: "944841e9d41e9b4deb34f2cece219afe"
server_key_version: "encryptionkey@0"
I20260926 16:37:17.898161 13316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "f88c2a8749bf49a8b92d4ba87bdd0a48"
format_stamp: "Formatted at 2026-09-26 16:37:17 on dist-test-slave-whjp"
server_key: "d93fefbcdd90fb84153d079039aa869b"
server_key_iv: "944841e9d41e9b4deb34f2cece219afe"
server_key_version: "encryptionkey@0"
I20260926 16:37:17.903196 13316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:37:17.906540 13331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:17.907945 13316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:17.908092 13316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "f88c2a8749bf49a8b92d4ba87bdd0a48"
format_stamp: "Formatted at 2026-09-26 16:37:17 on dist-test-slave-whjp"
server_key: "d93fefbcdd90fb84153d079039aa869b"
server_key_iv: "944841e9d41e9b4deb34f2cece219afe"
server_key_version: "encryptionkey@0"
I20260926 16:37:17.908274 13316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:17.933080 13316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:17.936798 13316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:17.937055 13316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:17.937969 13316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:17.939244 13316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:17.939320 13316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:17.939438 13316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:17.939489 13316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:17.967749 13316 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:33627
I20260926 16:37:17.967765 13443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:33627 every 8 connection(s)
I20260926 16:37:17.969220 13316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:17.971993 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13316
I20260926 16:37:17.972164 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:17.972574 10680 external_mini_cluster.cc:1428] Setting key f315c596f7bad1ae3f172dba1380acb1
Sep 26 16:37:17 dist-test-slave-whjp krb5kdc[13076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440637, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:17.990724 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:17.971679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:52537 (local address 127.10.110.62:42313)
0926 16:37:17.971887 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:17.971893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:17.973375 (+ 1482us) server_negotiation.cc:407] Connection header received
0926 16:37:17.975503 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:17.975509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:17.975590 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:17.975681 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:17.977175 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.977894 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.979424 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.979932 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.980038 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:17.984206 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:17.984212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:17.984217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:17.984254 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:17.987287 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:17.988187 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:17.988193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:17.988197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:17.988368 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:17.988914 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:17.988919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:17.988923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:17.989278 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:17.989435 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:17.990132 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 16:37:17.990366 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:37:17.991588 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:17.972356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:42313 (local address 127.10.110.2:52537)
0926 16:37:17.973222 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:17.973281 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:17.975244 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:17.975956 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:17.975964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:17.976462 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:17.977021 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:17.977039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.978048 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.978051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:17.979282 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:17.979295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:17.980150 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:17.980157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:17.980382 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:17.981125 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:17.981155 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:17.983957 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:17.987464 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:17.987472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:17.987493 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:17.988003 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:17.988510 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:17.988513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:17.988571 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:17.988789 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:17.989397 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:17.989403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:17.989758 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 16:37:17.989988 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 16:37:17.990409 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":339,"threads_started":1}
I20260926 16:37:17.993038 13444 heartbeater.cc:347] Connected to a master server at 127.10.110.62:42313
I20260926 16:37:17.993383 13444 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:37:17.994237 13444 heartbeater.cc:511] Master 127.10.110.62:42313 requested a full tablet report, sending...
I20260926 16:37:17.995811 13121 ts_manager.cc:194] Registered new tserver with Master: f88c2a8749bf49a8b92d4ba87bdd0a48 (127.10.110.2:33627)
I20260926 16:37:17.996634 13121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:52537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:18.015017 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:42313
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:18.326639 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:18.326923 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:18.327040 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:18.334379 13449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:18.334489 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:18.334641 13449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:18.344375 13449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:42313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:18.345654 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:18.346922 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:18.359433 13457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:18.359570 13455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:18.359477 13454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:18.360595 13449 server_base.cc:1034] running on GCE node
I20260926 16:37:18.361213 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:18.362207 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:18.363400 13449 hybrid_clock.cc:648] HybridClock initialized: now 1790440638363346 us; error 52 us; skew 500 ppm
I20260926 16:37:18.363701 13449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:18 dist-test-slave-whjp krb5kdc[13076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440638, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:18.370172 13449 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:18.373659 13449 webserver.cc:460] Webserver started at http://127.10.110.3:36083/ using document root <none> and password file <none>
I20260926 16:37:18.374465 13449 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:18.374569 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:18.374922 13449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:18.378610 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "d21811beb108422fa638f99beb16e9d8"
format_stamp: "Formatted at 2026-09-26 16:37:18 on dist-test-slave-whjp"
server_key: "967bcc513a8567bc2be9b93a55d59de6"
server_key_iv: "acbf545561d924e46a6592099b943466"
server_key_version: "encryptionkey@0"
I20260926 16:37:18.379381 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "d21811beb108422fa638f99beb16e9d8"
format_stamp: "Formatted at 2026-09-26 16:37:18 on dist-test-slave-whjp"
server_key: "967bcc513a8567bc2be9b93a55d59de6"
server_key_iv: "acbf545561d924e46a6592099b943466"
server_key_version: "encryptionkey@0"
I20260926 16:37:18.384493 13449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:37:18.387801 13464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:18.389108 13449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:37:18.389243 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "d21811beb108422fa638f99beb16e9d8"
format_stamp: "Formatted at 2026-09-26 16:37:18 on dist-test-slave-whjp"
server_key: "967bcc513a8567bc2be9b93a55d59de6"
server_key_iv: "acbf545561d924e46a6592099b943466"
server_key_version: "encryptionkey@0"
I20260926 16:37:18.389418 13449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:18.408377 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:18.412477 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:18.412761 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:18.413748 13449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:18.415192 13449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:18.415279 13449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:18.415375 13449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:18.415417 13449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:18.443742 13449 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38055
I20260926 16:37:18.443806 13576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38055 every 8 connection(s)
I20260926 16:37:18.445007 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:18.451502 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13449
I20260926 16:37:18.451680 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:18.452121 10680 external_mini_cluster.cc:1428] Setting key bc51e67b10af4d9601c393107fffb7cc
Sep 26 16:37:18 dist-test-slave-whjp krb5kdc[13076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440638, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:18.465492 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:18.447344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:42669 (local address 127.10.110.62:42313)
0926 16:37:18.447504 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:18.447509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:18.448728 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 16:37:18.450462 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:18.450467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:18.450535 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:18.450619 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:18.452175 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:18.453105 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:18.455106 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:18.455605 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:18.455701 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:18.459101 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:18.459114 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:18.459117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:18.459153 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:18.462159 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:18.463001 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:18.463006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:18.463009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:18.463155 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:18.463613 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:18.463617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:18.463619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:18.463967 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:18.464114 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:18.464849 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 16:37:18.465097 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:37:18.466750 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:18.447819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:42313 (local address 127.10.110.3:42669)
0926 16:37:18.448564 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:18.448603 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:18.450222 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:18.450817 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:18.450826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:18.451332 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:18.452000 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:18.452020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:18.453268 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:18.453273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:18.454978 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:18.454992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:18.455798 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:18.455803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:18.456050 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:18.456637 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:18.456662 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:18.458853 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:18.462327 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:18.462337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:18.462358 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:18.462853 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:18.463270 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:18.463276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:18.463280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:18.463481 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:18.464101 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:18.464109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:18.464464 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 16:37:18.464735 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 16:37:18.465278 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260926 16:37:18.468117 13577 heartbeater.cc:347] Connected to a master server at 127.10.110.62:42313
I20260926 16:37:18.468408 13577 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:18.469098 13577 heartbeater.cc:511] Master 127.10.110.62:42313 requested a full tablet report, sending...
I20260926 16:37:18.470409 13121 ts_manager.cc:194] Registered new tserver with Master: d21811beb108422fa638f99beb16e9d8 (127.10.110.3:38055)
I20260926 16:37:18.471163 13121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:42669
I20260926 16:37:18.479488 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:37:18.506028 13312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:18.503556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.10.110.62:42313)
0926 16:37:18.503755 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:18.503762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:18.504206 (+ 444us) server_negotiation.cc:407] Connection header received
0926 16:37:18.504368 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:18.504372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:18.504432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:18.504511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:18.505791 (+ 1280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60276: BlockingRecv error: recv got EOF from 127.0.0.1:60276 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:37:18.509395 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13182
W20260926 16:37:18.511801 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13182: Not authorized: Client connection negotiation failed: client connection to 127.10.110.1:33401: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:37:18.511970 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13182
I20260926 16:37:18.524389 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13316
W20260926 16:37:18.526726 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13316: Not authorized: Client connection negotiation failed: client connection to 127.10.110.2:33627: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:37:18.526889 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13316
I20260926 16:37:18.538805 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13449
W20260926 16:37:18.541489 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13449: Not authorized: Client connection negotiation failed: client connection to 127.10.110.3:38055: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:37:18.541594 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13449
I20260926 16:37:18.553368 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13089
I20260926 16:37:18.554476 13151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:18.655143 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13089
2026-09-26T16:37:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3245 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:18 dist-test-slave-whjp krb5kdc[13594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:18 dist-test-slave-whjp krb5kdc[13594](info): set up 2 sockets
Sep 26 16:37:18 dist-test-slave-whjp krb5kdc[13594](info): commencing operation
krb5kdc: starting...
W20260926 16:37:19.644719 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:19 dist-test-slave-whjp krb5kdc[13594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:19.831581 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:42473
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:20.139843 13607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:20.140131 13607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:20.140226 13607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:20.147380 13607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:20.147500 13607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:20.147560 13607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:20.147615 13607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:20.147661 13607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:20.157420 13607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:42473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:20.158659 13607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:20.159922 13607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:20.170862 13612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:20.170900 13615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:20.170924 13613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:20.171911 13607 server_base.cc:1034] running on GCE node
I20260926 16:37:20.172643 13607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:20.174003 13607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:20.175195 13607 hybrid_clock.cc:648] HybridClock initialized: now 1790440640175133 us; error 53 us; skew 500 ppm
I20260926 16:37:20.175495 13607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:20 dist-test-slave-whjp krb5kdc[13594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440640, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:20.178442 13607 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:20.179778 13607 webserver.cc:460] Webserver started at http://127.10.110.62:35179/ using document root <none> and password file <none>
I20260926 16:37:20.180461 13607 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:20.180560 13607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:20.180819 13607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:20.183670 13607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "eb9cdda6555b426399cdc5da5d0d8a43"
format_stamp: "Formatted at 2026-09-26 16:37:20 on dist-test-slave-whjp"
server_key: "9e73386ceb4885b3490cae44d7b2a66b"
server_key_iv: "56eb16c862e7881f4357f873d1cf8cf5"
server_key_version: "encryptionkey@0"
I20260926 16:37:20.184406 13607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "eb9cdda6555b426399cdc5da5d0d8a43"
format_stamp: "Formatted at 2026-09-26 16:37:20 on dist-test-slave-whjp"
server_key: "9e73386ceb4885b3490cae44d7b2a66b"
server_key_iv: "56eb16c862e7881f4357f873d1cf8cf5"
server_key_version: "encryptionkey@0"
I20260926 16:37:20.189307 13607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:37:20.192625 13622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:20.193934 13607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:37:20.194083 13607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "eb9cdda6555b426399cdc5da5d0d8a43"
format_stamp: "Formatted at 2026-09-26 16:37:20 on dist-test-slave-whjp"
server_key: "9e73386ceb4885b3490cae44d7b2a66b"
server_key_iv: "56eb16c862e7881f4357f873d1cf8cf5"
server_key_version: "encryptionkey@0"
I20260926 16:37:20.194250 13607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:20.218585 13607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:20.222673 13607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:20.222918 13607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:20.239720 13607 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:42473
I20260926 16:37:20.239749 13673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:42473 every 8 connection(s)
I20260926 16:37:20.241019 13607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:20.244498 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:20.250887 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13607
I20260926 16:37:20.251122 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:20.251904 10680 external_mini_cluster.cc:1428] Setting key b4591246c162af996326846efd988c41
I20260926 16:37:20.255191 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43: Bootstrap starting.
I20260926 16:37:20.258679 13674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:20.259755 13674 log.cc:824] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:20.262990 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43: No bootstrap required, opened a new log
Sep 26 16:37:20 dist-test-slave-whjp krb5kdc[13594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:20.271930 13674 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42473 } }
I20260926 16:37:20.272375 13674 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:20.272490 13674 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9cdda6555b426399cdc5da5d0d8a43, State: Initialized, Role: FOLLOWER
I20260926 16:37:20.273185 13674 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42473 } }
I20260926 16:37:20.273417 13674 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:20.273593 13674 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:20.273792 13674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:20.275620 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:20.253751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.10.110.62:42473)
0926 16:37:20.254462 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:20.254474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:20.254512 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:37:20.256309 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:20.256331 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:20.256755 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:20.257083 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:20.258821 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:20.260514 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:20.262064 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:20.263185 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:20.263390 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:20.267630 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:20.267638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:20.267655 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:20.267716 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:20.270983 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:20.271781 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:20.271788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:20.271794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:20.271986 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:20.272428 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:20.272433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:20.272436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:20.273226 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:20.273483 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:20.274005 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 16:37:20.274453 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":328,"threads_started":1}
I20260926 16:37:20.279230 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:37:20.313740 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:20.306368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.10.110.62:42473)
0926 16:37:20.306588 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:20.306595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:20.306981 (+ 386us) server_negotiation.cc:407] Connection header received
0926 16:37:20.307253 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:20.307257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:20.307325 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:20.307420 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:20.308493 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:20.309186 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:20.310516 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:20.310965 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:20.311097 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:20.312258 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:20.312264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:20.312267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:20.312953 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:20.313017 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:20.313048 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:20.313157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:20.313295 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 16:37:20.313401 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:20.331995 13674 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42473 } }
I20260926 16:37:20.332700 13674 leader_election.cc:302] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9cdda6555b426399cdc5da5d0d8a43; no voters:
I20260926 16:37:20.333371 13674 leader_election.cc:288] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:20.333489 13686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:20.336145 13686 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [term 1 LEADER]: Becoming Leader. State: Replica: eb9cdda6555b426399cdc5da5d0d8a43, State: Running, Role: LEADER
I20260926 16:37:20.336786 13686 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42473 } }
I20260926 16:37:20.337754 13674 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:20.339332 13687 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb9cdda6555b426399cdc5da5d0d8a43. Latest consensus state: current_term: 1 leader_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42473 } } }
I20260926 16:37:20.339902 13687 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:20.339819 13688 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9cdda6555b426399cdc5da5d0d8a43" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42473 } } }
I20260926 16:37:20.340171 13688 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:20.341847 13691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:20.346827 13691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:20.354537 13691 catalog_manager.cc:1347] Generated new cluster ID: fa90f8d048434a07935f820d9d4cdab0
I20260926 16:37:20.354650 13691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:20.389088 13691 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:20.390959 13691 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:20.404011 13691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb9cdda6555b426399cdc5da5d0d8a43: Generated new TSK 0
I20260926 16:37:20.404958 13691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:37:20.422963 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13607
I20260926 16:37:20.424304 13669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:20.515136 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13607
2026-09-26T16:37:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1855 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:21 dist-test-slave-whjp krb5kdc[13708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:21 dist-test-slave-whjp krb5kdc[13708](info): set up 2 sockets
Sep 26 16:37:21 dist-test-slave-whjp krb5kdc[13708](info): commencing operation
krb5kdc: starting...
W20260926 16:37:22.512763 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:22 dist-test-slave-whjp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:22.685237 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44827
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:44827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:22.992933 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:22.993222 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:22.993317 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:23.000486 13721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:23.000604 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:23.000656 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:23.000718 13721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:23.000761 13721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:23.010604 13721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:44827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:23.011891 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:23.013142 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:23.026557 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:23.026598 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:23.026525 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:23.027479 13721 server_base.cc:1034] running on GCE node
I20260926 16:37:23.028218 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:23.029505 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:23.030711 13721 hybrid_clock.cc:648] HybridClock initialized: now 1790440643030661 us; error 39 us; skew 500 ppm
I20260926 16:37:23.031023 13721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:23 dist-test-slave-whjp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440643, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:23.033979 13721 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:23.035372 13721 webserver.cc:460] Webserver started at http://127.10.110.62:45135/ using document root <none> and password file <none>
I20260926 16:37:23.036062 13721 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:23.036175 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:23.036437 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:23.039194 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "44125f3e9d8c476bb57c6916dccf4ae5"
format_stamp: "Formatted at 2026-09-26 16:37:23 on dist-test-slave-whjp"
server_key: "b5ddfd74d803dbcd6cbd934fd3e0cf6c"
server_key_iv: "a4cac0c448ac1a780ca52cf85a89b863"
server_key_version: "encryptionkey@0"
I20260926 16:37:23.039937 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "44125f3e9d8c476bb57c6916dccf4ae5"
format_stamp: "Formatted at 2026-09-26 16:37:23 on dist-test-slave-whjp"
server_key: "b5ddfd74d803dbcd6cbd934fd3e0cf6c"
server_key_iv: "a4cac0c448ac1a780ca52cf85a89b863"
server_key_version: "encryptionkey@0"
I20260926 16:37:23.044654 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:37:23.047870 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:23.049094 13721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:23.049237 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "44125f3e9d8c476bb57c6916dccf4ae5"
format_stamp: "Formatted at 2026-09-26 16:37:23 on dist-test-slave-whjp"
server_key: "b5ddfd74d803dbcd6cbd934fd3e0cf6c"
server_key_iv: "a4cac0c448ac1a780ca52cf85a89b863"
server_key_version: "encryptionkey@0"
I20260926 16:37:23.049394 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:23.075915 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:23.079864 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:23.080127 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:23.097316 13721 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:44827
I20260926 16:37:23.097350 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:44827 every 8 connection(s)
I20260926 16:37:23.098613 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:23.101063 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13721
I20260926 16:37:23.101281 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:23.102032 10680 external_mini_cluster.cc:1428] Setting key 9ff7d75ef229f1e74697b965f9cae546
I20260926 16:37:23.102885 13788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:37:23 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:23.117239 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5: Bootstrap starting.
I20260926 16:37:23.120484 13788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:23.121711 13788 log.cc:824] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:23.124114 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:23.104104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.10.110.62:44827)
0926 16:37:23.104853 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:23.104866 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:23.104907 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:37:23.106376 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:23.106396 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:23.106799 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:23.107126 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:23.108806 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:23.110125 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:23.111819 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:23.112513 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:23.112638 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:23.117274 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:23.117283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:23.117299 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:23.117336 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:23.120120 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:23.120905 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:23.120911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:23.120917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:23.121080 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:23.121466 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:23.121469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:23.121471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:23.121983 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:23.122242 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:23.122711 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 16:37:23.123109 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":322,"threads_started":1}
I20260926 16:37:23.124678 13788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5: No bootstrap required, opened a new log
I20260926 16:37:23.132170 13788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44827 } }
I20260926 16:37:23.132521 13788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:23.132606 13788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44125f3e9d8c476bb57c6916dccf4ae5, State: Initialized, Role: FOLLOWER
I20260926 16:37:23.133164 13788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44827 } }
I20260926 16:37:23.133436 13788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:23.133618 13788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:23.133819 13788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:23.135988 13788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44827 } }
I20260926 16:37:23.136440 13788 leader_election.cc:302] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44125f3e9d8c476bb57c6916dccf4ae5; no voters:
I20260926 16:37:23.136929 13788 leader_election.cc:288] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:23.137045 13793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:23.137645 13793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 44125f3e9d8c476bb57c6916dccf4ae5, State: Running, Role: LEADER
I20260926 16:37:23.138157 13793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44827 } }
I20260926 16:37:23.139065 13788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:23.140620 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44125f3e9d8c476bb57c6916dccf4ae5. Latest consensus state: current_term: 1 leader_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44827 } } }
I20260926 16:37:23.140889 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:23.141283 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44125f3e9d8c476bb57c6916dccf4ae5" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44827 } } }
I20260926 16:37:23.141582 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:23.142424 13799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:23.147066 13799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:23.156248 13799 catalog_manager.cc:1347] Generated new cluster ID: 8fed5afe600c4f76899e81ee5ed6635e
I20260926 16:37:23.156353 13799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:23.173096 13799 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:23.174868 13799 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:23.183498 13799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44125f3e9d8c476bb57c6916dccf4ae5: Generated new TSK 0
I20260926 16:37:23.184393 13799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:23.238489 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44827
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:23.564509 13814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:23.564785 13814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:23.564904 13814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:23.572002 13814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:23.572120 13814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:23.572371 13814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:23.582003 13814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:23.583395 13814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:23.584705 13814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:23.596849 13820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:23.596820 13819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:23.596925 13822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:23.597764 13814 server_base.cc:1034] running on GCE node
I20260926 16:37:23.598358 13814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:23.599243 13814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:23.600425 13814 hybrid_clock.cc:648] HybridClock initialized: now 1790440643600374 us; error 42 us; skew 500 ppm
I20260926 16:37:23.600704 13814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:23 dist-test-slave-whjp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440643, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:23.612238 13814 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:23.613750 13814 webserver.cc:460] Webserver started at http://127.10.110.1:38391/ using document root <none> and password file <none>
I20260926 16:37:23.614426 13814 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:23.614532 13814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:23.614786 13814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:23.617677 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "a8566ca25e9742e9b3013df6cca40992"
format_stamp: "Formatted at 2026-09-26 16:37:23 on dist-test-slave-whjp"
server_key: "d86c6a1b7a07e62c6c0de8d55657d74b"
server_key_iv: "dfa648068862667bb1cd6c4812fc1e41"
server_key_version: "encryptionkey@0"
I20260926 16:37:23.618389 13814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "a8566ca25e9742e9b3013df6cca40992"
format_stamp: "Formatted at 2026-09-26 16:37:23 on dist-test-slave-whjp"
server_key: "d86c6a1b7a07e62c6c0de8d55657d74b"
server_key_iv: "dfa648068862667bb1cd6c4812fc1e41"
server_key_version: "encryptionkey@0"
I20260926 16:37:23.623548 13814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:37:23.626940 13829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:23.628293 13814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:23.628435 13814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "a8566ca25e9742e9b3013df6cca40992"
format_stamp: "Formatted at 2026-09-26 16:37:23 on dist-test-slave-whjp"
server_key: "d86c6a1b7a07e62c6c0de8d55657d74b"
server_key_iv: "dfa648068862667bb1cd6c4812fc1e41"
server_key_version: "encryptionkey@0"
I20260926 16:37:23.628612 13814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:23.654078 13814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:23.658210 13814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:23.658476 13814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:23.659404 13814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:23.660737 13814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:23.660815 13814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:23.660930 13814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:23.660980 13814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:23.689637 13814 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:41465
I20260926 16:37:23.689678 13941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:41465 every 8 connection(s)
I20260926 16:37:23.690855 13814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:23.697647 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13814
I20260926 16:37:23.697830 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:23.698441 10680 external_mini_cluster.cc:1428] Setting key f2464031502dcc064627c2ff7c7dfd61
Sep 26 16:37:23 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440643, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:23.709456 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:23.693227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:45745 (local address 127.10.110.62:44827)
0926 16:37:23.693635 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:23.693639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:23.694601 (+ 962us) server_negotiation.cc:407] Connection header received
0926 16:37:23.696406 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:23.696411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:23.696490 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:23.696581 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:23.698141 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:23.698865 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:23.700470 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:23.700832 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:23.700938 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:23.704184 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:23.704189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:23.704193 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:23.704222 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:23.706616 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:23.707310 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:23.707314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:23.707316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:23.707422 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:23.707812 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:23.707816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:23.707818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:23.708095 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:23.708227 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:23.708901 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 16:37:23.709113 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":196,"threads_started":1}
I20260926 16:37:23.710469 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:23.693822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44827 (local address 127.10.110.1:45745)
0926 16:37:23.694460 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:23.694496 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:23.696148 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:23.696836 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:23.696845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:23.697349 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:23.697982 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:23.698001 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:23.699028 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:23.699032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:23.700327 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:23.700340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:23.701039 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:23.701046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:23.701263 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:23.701862 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:23.701884 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:23.703985 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:23.706769 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:23.706778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:23.706795 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:23.707191 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:23.707515 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:23.707520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:23.707524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:23.707705 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:23.708216 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:23.708223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:23.708542 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 16:37:23.708777 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 16:37:23.709314 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":241,"threads_started":1}
I20260926 16:37:23.711925 13942 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44827
I20260926 16:37:23.712253 13942 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:23.713122 13942 heartbeater.cc:511] Master 127.10.110.62:44827 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:37:23.715270 13753 ts_manager.cc:194] Registered new tserver with Master: a8566ca25e9742e9b3013df6cca40992 (127.10.110.1:41465)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:37:23.716823 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:45745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:23.734656 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44827
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:24.038663 13948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:24.038947 13948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:24.039103 13948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:24.046391 13948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:24.046509 13948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:24.046674 13948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:24.056408 13948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:24.057876 13948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:24.059530 13948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:24.072355 13954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:24.072661 13956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:24.072469 13953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:24.072846 13948 server_base.cc:1034] running on GCE node
I20260926 16:37:24.073516 13948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:24.074417 13948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:24.075611 13948 hybrid_clock.cc:648] HybridClock initialized: now 1790440644075553 us; error 59 us; skew 500 ppm
I20260926 16:37:24.075933 13948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:24.078899 13948 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:24.080240 13948 webserver.cc:460] Webserver started at http://127.10.110.2:36547/ using document root <none> and password file <none>
I20260926 16:37:24.080892 13948 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:24.081007 13948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:24.081281 13948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:24.084710 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "114a083949474b9d86c1a1e2f653f86c"
format_stamp: "Formatted at 2026-09-26 16:37:24 on dist-test-slave-whjp"
server_key: "2761a7c3928c49439c1b89bb8e6192b9"
server_key_iv: "81d54e12794c219a2dfc8c3f23b0a952"
server_key_version: "encryptionkey@0"
I20260926 16:37:24.085489 13948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "114a083949474b9d86c1a1e2f653f86c"
format_stamp: "Formatted at 2026-09-26 16:37:24 on dist-test-slave-whjp"
server_key: "2761a7c3928c49439c1b89bb8e6192b9"
server_key_iv: "81d54e12794c219a2dfc8c3f23b0a952"
server_key_version: "encryptionkey@0"
I20260926 16:37:24.090561 13948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:37:24.093916 13963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:24.095302 13948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:24.095446 13948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "114a083949474b9d86c1a1e2f653f86c"
format_stamp: "Formatted at 2026-09-26 16:37:24 on dist-test-slave-whjp"
server_key: "2761a7c3928c49439c1b89bb8e6192b9"
server_key_iv: "81d54e12794c219a2dfc8c3f23b0a952"
server_key_version: "encryptionkey@0"
I20260926 16:37:24.095628 13948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:24.109658 13948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:24.113579 13948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:24.113839 13948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:24.114738 13948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:24.116001 13948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:24.116076 13948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:24.116190 13948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:24.116252 13948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:24.144933 13948 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:40013
I20260926 16:37:24.144973 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:40013 every 8 connection(s)
I20260926 16:37:24.146158 13948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:24.149504 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 13948
I20260926 16:37:24.149655 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:24.150071 10680 external_mini_cluster.cc:1428] Setting key 0d4b8de9b8a66369b631a391a44bb893
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:24.165959 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.148603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:58429 (local address 127.10.110.62:44827)
0926 16:37:24.148755 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:24.148759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:24.150103 (+ 1344us) server_negotiation.cc:407] Connection header received
0926 16:37:24.152290 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:24.152296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:24.152377 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:24.152474 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:24.153886 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.154844 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.156398 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.156909 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.157000 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.160681 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:24.160685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:24.160689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:24.160719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:24.163106 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.163776 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.163780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.163782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.163936 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.164335 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.164341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.164343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.164592 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:24.164737 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:24.165405 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 16:37:24.165621 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:37:24.166945 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.149171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44827 (local address 127.10.110.2:58429)
0926 16:37:24.149929 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:24.149980 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:24.152049 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:24.152645 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:24.152653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:24.153122 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:24.153737 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:24.153756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.154991 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.154994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:24.156277 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:24.156288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.157107 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.157113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:24.157390 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.158055 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:24.158082 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:24.160463 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:24.163243 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:24.163252 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:24.163269 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:24.163659 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:24.164034 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:24.164040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:24.164044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:24.164229 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:24.164707 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:24.164715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:24.165043 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 16:37:24.165291 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 16:37:24.165811 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":293,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:37:24.168324 14076 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44827
I20260926 16:37:24.168597 14076 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:24.169296 14076 heartbeater.cc:511] Master 127.10.110.62:44827 requested a full tablet report, sending...
I20260926 16:37:24.170661 13753 ts_manager.cc:194] Registered new tserver with Master: 114a083949474b9d86c1a1e2f653f86c (127.10.110.2:40013)
I20260926 16:37:24.171655 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:58429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:24.188282 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44827
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:24.500186 14081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:24.500465 14081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:24.500562 14081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:24.508352 14081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:24.508472 14081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:24.508584 14081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:24.518267 14081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:24.519452 14081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:24.520694 14081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:24.532898 14086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:24.532840 14089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:24.532814 14087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:24.533586 14081 server_base.cc:1034] running on GCE node
I20260926 16:37:24.534207 14081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:24.535089 14081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:24.536288 14081 hybrid_clock.cc:648] HybridClock initialized: now 1790440644536234 us; error 44 us; skew 500 ppm
I20260926 16:37:24.536597 14081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:24.539685 14081 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:24.541070 14081 webserver.cc:460] Webserver started at http://127.10.110.3:33561/ using document root <none> and password file <none>
I20260926 16:37:24.541716 14081 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:24.541821 14081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:24.542062 14081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:24.545017 14081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "3595f3a233204b9ea77e03fa7b5fb325"
format_stamp: "Formatted at 2026-09-26 16:37:24 on dist-test-slave-whjp"
server_key: "9e926efa9eadbcc3f70eb574f97cc4d5"
server_key_iv: "9d65fab2c3183a6939091cfd9a130f93"
server_key_version: "encryptionkey@0"
I20260926 16:37:24.545753 14081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "3595f3a233204b9ea77e03fa7b5fb325"
format_stamp: "Formatted at 2026-09-26 16:37:24 on dist-test-slave-whjp"
server_key: "9e926efa9eadbcc3f70eb574f97cc4d5"
server_key_iv: "9d65fab2c3183a6939091cfd9a130f93"
server_key_version: "encryptionkey@0"
I20260926 16:37:24.551030 14081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 16:37:24.554600 14096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:24.556062 14081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:37:24.556216 14081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "3595f3a233204b9ea77e03fa7b5fb325"
format_stamp: "Formatted at 2026-09-26 16:37:24 on dist-test-slave-whjp"
server_key: "9e926efa9eadbcc3f70eb574f97cc4d5"
server_key_iv: "9d65fab2c3183a6939091cfd9a130f93"
server_key_version: "encryptionkey@0"
I20260926 16:37:24.556399 14081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:24.594261 14081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:24.598652 14081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:24.598973 14081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:24.600059 14081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:24.601390 14081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:24.601475 14081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:24.601599 14081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:24.601655 14081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:24.631021 14081 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:39711
I20260926 16:37:24.631067 14208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:39711 every 8 connection(s)
I20260926 16:37:24.632344 14081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:24.635601 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14081
I20260926 16:37:24.635764 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:24.636196 10680 external_mini_cluster.cc:1428] Setting key b4b844d0b48796e9dd249f5ed356eeff
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:24.651551 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.634761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:37635 (local address 127.10.110.62:44827)
0926 16:37:24.634970 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:24.634977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:24.636293 (+ 1316us) server_negotiation.cc:407] Connection header received
0926 16:37:24.638307 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:24.638312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:24.638395 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:24.638475 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:24.640227 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.640935 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.642492 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.642881 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.642980 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.646308 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:24.646313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:24.646316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:24.646350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:24.648723 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.649424 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.649428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.649430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.649535 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.649942 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.649946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.649948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.650211 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:24.650335 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:24.651009 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 16:37:24.651224 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:37:24.652622 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.635323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44827 (local address 127.10.110.3:37635)
0926 16:37:24.636099 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:24.636146 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:24.638051 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:24.638792 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:24.638801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:24.639332 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:24.640060 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:24.640082 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.641077 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.641083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:24.642356 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:24.642368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.643088 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.643095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:24.643322 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.643993 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:24.644019 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:24.646099 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:24.648868 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:24.648875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:24.648889 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:24.649274 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:24.649641 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:24.649645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:24.649649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:24.649835 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:24.650328 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:24.650335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:24.650648 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 16:37:24.650885 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 16:37:24.651403 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":305,"threads_started":1}
I20260926 16:37:24.653877 14209 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44827
I20260926 16:37:24.654181 14209 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:24.654919 14209 heartbeater.cc:511] Master 127.10.110.62:44827 requested a full tablet report, sending...
I20260926 16:37:24.656291 13753 ts_manager.cc:194] Registered new tserver with Master: 3595f3a233204b9ea77e03fa7b5fb325 (127.10.110.3:39711)
I20260926 16:37:24.657032 13753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:37635
I20260926 16:37:24.662993 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:24.705250 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.692724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.10.110.62:44827)
0926 16:37:24.692916 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:24.692922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:24.693227 (+ 305us) server_negotiation.cc:407] Connection header received
0926 16:37:24.693425 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:24.693429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:24.693490 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:24.693562 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:24.694983 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.695740 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.696884 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.697279 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.697365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.700979 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:24.700983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:24.700986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:24.701019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:24.703151 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.703706 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.703710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.703712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.703867 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.704145 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.704148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.704150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.704457 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:24.704624 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:24.704829 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 16:37:24.704973 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 16:37:24.705792 13753 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36974
I20260926 16:37:24.707193 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13814
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:24.719481 13942 heartbeater.cc:503] Master 127.10.110.62:44827 was elected leader, sending a full tablet report...
I20260926 16:37:24.721669 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.708215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.10.110.1:41465)
0926 16:37:24.708733 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:24.708740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:24.708771 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:37:24.708822 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:24.708828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:24.709028 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:24.709148 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:24.710123 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.711091 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.712232 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.712780 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.712870 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.716164 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:24.716174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:24.716188 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:24.716218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:24.718692 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.719389 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.719397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.719400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.719547 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.720186 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.720190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.720192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.720524 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:24.720799 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:24.721109 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 16:37:24.721359 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
W20260926 16:37:24.722808 13937 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60766
W20260926 16:37:24.723877 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13814: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:24.724059 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13814
I20260926 16:37:24.736737 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13948
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:24.751780 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.737490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.10.110.2:40013)
0926 16:37:24.738097 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:24.738105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:24.738132 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:37:24.738195 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:24.738201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:24.738498 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:24.738674 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:24.739920 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.740940 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.742174 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.742721 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.742834 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.746585 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:24.746599 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:24.746614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:24.746647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:24.749260 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.749861 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.749868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.749870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.749980 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.750317 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.750322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.750326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.750640 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:24.750913 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:24.751216 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 16:37:24.751463 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":267,"threads_started":1}
W20260926 16:37:24.753095 14071 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59768
W20260926 16:37:24.754220 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13948: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:24.754350 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13948
I20260926 16:37:24.766491 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14081
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440644, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:24.781766 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:24.767390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.10.110.3:39711)
0926 16:37:24.767991 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:24.767998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:24.768028 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:37:24.768085 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:24.768091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:24.768308 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:24.768448 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:24.769697 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.770729 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.771961 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:24.772507 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:24.772617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:24.776570 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:24.776581 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:24.776595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:24.776631 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:24.779032 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.779645 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.779652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.779654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.779769 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:24.780251 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:24.780256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:24.780258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:24.780589 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:24.780880 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:24.781168 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 16:37:24.781434 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":305,"threads_started":1}
W20260926 16:37:24.782949 14204 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32998
W20260926 16:37:24.783994 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14081: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:37:24.784143 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14081
I20260926 16:37:24.795940 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 13721
I20260926 16:37:24.796990 13783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:24.898187 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 13721
2026-09-26T16:37:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4388 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[14226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[14226](info): set up 2 sockets
Sep 26 16:37:24 dist-test-slave-whjp krb5kdc[14226](info): commencing operation
krb5kdc: starting...
W20260926 16:37:25.897089 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:26 dist-test-slave-whjp krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:26.077402 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:41583
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:26.386808 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:26.387106 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:26.387295 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:26.394429 14239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:26.394542 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:26.394618 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:26.394691 14239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:26.394748 14239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:26.404484 14239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:26.405755 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:26.407004 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:26.419299 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:26.419369 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:26.419341 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:26.420405 14239 server_base.cc:1034] running on GCE node
I20260926 16:37:26.421099 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:26.422466 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:26.423687 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790440646423637 us; error 46 us; skew 500 ppm
I20260926 16:37:26.424029 14239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:26 dist-test-slave-whjp krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440646, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:26.427182 14239 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:26.428578 14239 webserver.cc:460] Webserver started at http://127.10.110.62:37523/ using document root <none> and password file <none>
I20260926 16:37:26.429222 14239 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:26.429332 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:26.429603 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:26.432502 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "abdd1a984bec47e98a8911314abfcef8"
format_stamp: "Formatted at 2026-09-26 16:37:26 on dist-test-slave-whjp"
server_key: "a7bfe82e75455e7c57c8f188afa7c79b"
server_key_iv: "b379b97cdd079f2f0c1ad97de8a89730"
server_key_version: "encryptionkey@0"
I20260926 16:37:26.433219 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "abdd1a984bec47e98a8911314abfcef8"
format_stamp: "Formatted at 2026-09-26 16:37:26 on dist-test-slave-whjp"
server_key: "a7bfe82e75455e7c57c8f188afa7c79b"
server_key_iv: "b379b97cdd079f2f0c1ad97de8a89730"
server_key_version: "encryptionkey@0"
I20260926 16:37:26.438104 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:37:26.441325 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:26.442576 14239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:37:26.442744 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "abdd1a984bec47e98a8911314abfcef8"
format_stamp: "Formatted at 2026-09-26 16:37:26 on dist-test-slave-whjp"
server_key: "a7bfe82e75455e7c57c8f188afa7c79b"
server_key_iv: "b379b97cdd079f2f0c1ad97de8a89730"
server_key_version: "encryptionkey@0"
I20260926 16:37:26.442906 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:26.462055 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:26.466114 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:26.466372 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:26.482023 14239 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:41583
I20260926 16:37:26.482033 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:41583 every 8 connection(s)
I20260926 16:37:26.483369 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:26.483705 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14239
I20260926 16:37:26.483913 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:26.484654 10680 external_mini_cluster.cc:1428] Setting key 8d95c2045f6f74567de2dba2858dedb1
I20260926 16:37:26.487367 14306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:37:26 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:26.499645 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8: Bootstrap starting.
I20260926 16:37:26.503616 14306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:26.504827 14306 log.cc:824] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:26.508090 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8: No bootstrap required, opened a new log
I20260926 16:37:26.508137 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:26.486602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.10.110.62:41583)
0926 16:37:26.487505 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:26.487521 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:26.487905 (+ 384us) server_negotiation.cc:407] Connection header received
0926 16:37:26.490076 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:26.490103 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:26.490586 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:26.491022 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:26.492235 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:26.494222 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:26.495348 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:26.496402 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:26.496598 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:26.500147 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:26.500157 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:26.500176 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:26.500219 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:26.503763 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:26.504431 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:26.504439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:26.504448 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:26.504678 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:26.504997 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:26.505002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:26.505005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:26.505672 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:26.505897 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:26.506379 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 16:37:26.506821 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":345,"threads_started":1}
I20260926 16:37:26.516384 14306 raft_consensus.cc:348] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdd1a984bec47e98a8911314abfcef8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41583 } }
I20260926 16:37:26.516834 14306 raft_consensus.cc:374] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:26.516955 14306 raft_consensus.cc:729] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdd1a984bec47e98a8911314abfcef8, State: Initialized, Role: FOLLOWER
I20260926 16:37:26.517527 14306 consensus_queue.cc:260] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdd1a984bec47e98a8911314abfcef8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41583 } }
I20260926 16:37:26.517742 14306 raft_consensus.cc:388] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:26.517859 14306 raft_consensus.cc:482] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:26.518007 14306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:26.520107 14306 raft_consensus.cc:504] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdd1a984bec47e98a8911314abfcef8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41583 } }
I20260926 16:37:26.520577 14306 leader_election.cc:302] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abdd1a984bec47e98a8911314abfcef8; no voters:
I20260926 16:37:26.521086 14306 leader_election.cc:288] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:26.521219 14311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:26.521816 14311 raft_consensus.cc:686] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [term 1 LEADER]: Becoming Leader. State: Replica: abdd1a984bec47e98a8911314abfcef8, State: Running, Role: LEADER
I20260926 16:37:26.522300 14311 consensus_queue.cc:237] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdd1a984bec47e98a8911314abfcef8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41583 } }
I20260926 16:37:26.522955 14306 sys_catalog.cc:564] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:26.524778 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abdd1a984bec47e98a8911314abfcef8. Latest consensus state: current_term: 1 leader_uuid: "abdd1a984bec47e98a8911314abfcef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdd1a984bec47e98a8911314abfcef8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41583 } } }
I20260926 16:37:26.525127 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:26.526232 14312 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abdd1a984bec47e98a8911314abfcef8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdd1a984bec47e98a8911314abfcef8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41583 } } }
I20260926 16:37:26.526598 14312 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:26.527786 14319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:26.532814 14319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:26.540822 14319 catalog_manager.cc:1347] Generated new cluster ID: 6e00d3d8d2e3487b8fecfe786dfde674
I20260926 16:37:26.540923 14319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:26.553879 14319 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:26.555352 14319 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:26.561625 14319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abdd1a984bec47e98a8911314abfcef8: Generated new TSK 0
I20260926 16:37:26.562523 14319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:26.611352 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41583
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:26.924342 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:26.924604 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:26.924705 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:26.931733 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:26.931852 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:26.931985 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:26.941581 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:26.942831 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:26.944069 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:26.956423 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:26.956346 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:26.956380 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:26.957136 14332 server_base.cc:1034] running on GCE node
I20260926 16:37:26.957710 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:26.958657 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:26.959884 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790440646959818 us; error 52 us; skew 500 ppm
I20260926 16:37:26.960238 14332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:26 dist-test-slave-whjp krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440646, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:26.963454 14332 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:26.964931 14332 webserver.cc:460] Webserver started at http://127.10.110.1:39399/ using document root <none> and password file <none>
I20260926 16:37:26.965608 14332 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:26.965721 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:26.966004 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:26.969094 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "f7210759a9314614877bbb5ecc5fc8fe"
format_stamp: "Formatted at 2026-09-26 16:37:26 on dist-test-slave-whjp"
server_key: "8ff53895ce289d4164e17def29ad8981"
server_key_iv: "bdb7607fd624aff91b7754998f2be63a"
server_key_version: "encryptionkey@0"
I20260926 16:37:26.970047 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "f7210759a9314614877bbb5ecc5fc8fe"
format_stamp: "Formatted at 2026-09-26 16:37:26 on dist-test-slave-whjp"
server_key: "8ff53895ce289d4164e17def29ad8981"
server_key_iv: "bdb7607fd624aff91b7754998f2be63a"
server_key_version: "encryptionkey@0"
I20260926 16:37:26.975376 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:37:26.979123 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:26.980635 14332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:26.980791 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "f7210759a9314614877bbb5ecc5fc8fe"
format_stamp: "Formatted at 2026-09-26 16:37:26 on dist-test-slave-whjp"
server_key: "8ff53895ce289d4164e17def29ad8981"
server_key_iv: "bdb7607fd624aff91b7754998f2be63a"
server_key_version: "encryptionkey@0"
I20260926 16:37:26.980980 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:26.993468 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:26.997303 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:26.997581 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:26.998487 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:26.999780 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:26.999881 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.000001 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:27.000063 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.028548 14332 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:35155
I20260926 16:37:27.028604 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:35155 every 8 connection(s)
I20260926 16:37:27.029846 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:27.038519 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14332
I20260926 16:37:27.038735 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:27.039456 10680 external_mini_cluster.cc:1428] Setting key a5df12bfe402b76b4ecb57c50387a3ab
Sep 26 16:37:27 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440646, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:27.048344 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:27.032031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:36955 (local address 127.10.110.62:41583)
0926 16:37:27.032458 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:27.032465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:27.033465 (+ 1000us) server_negotiation.cc:407] Connection header received
0926 16:37:27.035177 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:27.035182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:27.035249 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:27.035337 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:27.036758 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.037417 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.039268 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.039691 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.039781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:27.043175 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:27.043180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:27.043190 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:27.043219 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:27.045499 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:27.046181 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:27.046209 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:27.046211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:27.046324 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:27.046707 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:27.046710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:27.046712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:27.046974 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:27.047116 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:27.047757 (+ 641us) server_negotiation.cc:299] Negotiation successful
0926 16:37:27.047984 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260926 16:37:27.049223 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:27.032616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41583 (local address 127.10.110.1:36955)
0926 16:37:27.033306 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:27.033352 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:27.034950 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:27.035512 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:27.035520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:27.036075 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:27.036628 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:27.036644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.037527 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.037531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:27.039091 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:27.039106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.039918 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.039926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:27.040153 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:27.040731 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:27.040760 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:27.042981 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:27.045628 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:27.045636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:27.045652 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:27.046074 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:27.046417 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:27.046422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:27.046426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:27.046602 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:27.047124 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:27.047132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:27.047436 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 16:37:27.047649 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 16:37:27.048179 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":237,"threads_started":1}
I20260926 16:37:27.050741 14460 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41583
I20260926 16:37:27.051038 14460 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:27.051743 14460 heartbeater.cc:511] Master 127.10.110.62:41583 requested a full tablet report, sending...
I20260926 16:37:27.053735 14271 ts_manager.cc:194] Registered new tserver with Master: f7210759a9314614877bbb5ecc5fc8fe (127.10.110.1:35155)
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:37:27.055192 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:36955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:27.077126 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41583
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:27.384202 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:27.384493 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:27.384614 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:27.391749 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:27.391880 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:27.391993 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:27.401439 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:27.402616 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:27.403873 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:27.416292 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:27.416253 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:27.416425 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:27.416993 14466 server_base.cc:1034] running on GCE node
I20260926 16:37:27.417528 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:27.418435 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:27.419623 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790440647419591 us; error 44 us; skew 500 ppm
I20260926 16:37:27.419950 14466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:27 dist-test-slave-whjp krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440647, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:27.422782 14466 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:27.424151 14466 webserver.cc:460] Webserver started at http://127.10.110.2:36457/ using document root <none> and password file <none>
I20260926 16:37:27.424770 14466 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:27.424880 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:27.425150 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:27.427875 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "de04e6e896ef4032b99eddd02deed4ba"
format_stamp: "Formatted at 2026-09-26 16:37:27 on dist-test-slave-whjp"
server_key: "6618a9acfd8a9470f0e3024ff456e399"
server_key_iv: "91d457a836ffebf863eea7e138d6318b"
server_key_version: "encryptionkey@0"
I20260926 16:37:27.428618 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "de04e6e896ef4032b99eddd02deed4ba"
format_stamp: "Formatted at 2026-09-26 16:37:27 on dist-test-slave-whjp"
server_key: "6618a9acfd8a9470f0e3024ff456e399"
server_key_iv: "91d457a836ffebf863eea7e138d6318b"
server_key_version: "encryptionkey@0"
I20260926 16:37:27.433475 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:37:27.436584 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.437809 14466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:37:27.437943 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "de04e6e896ef4032b99eddd02deed4ba"
format_stamp: "Formatted at 2026-09-26 16:37:27 on dist-test-slave-whjp"
server_key: "6618a9acfd8a9470f0e3024ff456e399"
server_key_iv: "91d457a836ffebf863eea7e138d6318b"
server_key_version: "encryptionkey@0"
I20260926 16:37:27.438110 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:27.453840 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:27.457705 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:27.457966 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:27.458940 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:27.460410 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:27.460486 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.460580 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:27.460637 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.491098 14466 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:36793
I20260926 16:37:27.491117 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:36793 every 8 connection(s)
I20260926 16:37:27.492357 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:27.492849 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14466
I20260926 16:37:27.493007 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:27.493481 10680 external_mini_cluster.cc:1428] Setting key 4c328386d7a0be5adac92865de7cc9b3
Sep 26 16:37:27 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440647, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:37:27.516032 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:27.496293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:35919 (local address 127.10.110.62:41583)
0926 16:37:27.496457 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:27.496463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:27.498215 (+ 1752us) server_negotiation.cc:407] Connection header received
0926 16:37:27.500913 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:27.500919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:27.501000 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:27.501102 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:27.503035 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.503737 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.505848 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.506242 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.506328 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:27.509961 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:27.509968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:27.509973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:27.510020 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:27.513083 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:27.513788 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:27.513792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:27.513794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:27.513914 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:27.514315 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:27.514319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:27.514321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:27.514563 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:27.514713 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:27.515416 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 16:37:27.515668 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 16:37:27.516956 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:27.495731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41583 (local address 127.10.110.2:35919)
0926 16:37:27.498067 (+ 2336us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:27.498112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:27.500612 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:27.501474 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:27.501487 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:27.502252 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:27.502872 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:27.502891 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.503916 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.503920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:27.505718 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:27.505733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.506441 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.506447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:27.506717 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:27.507375 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:27.507402 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:27.509731 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:27.513228 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:27.513235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:27.513247 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:27.513669 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:27.513991 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:27.513995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:27.513997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:27.514183 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:27.514680 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:27.514687 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:27.515026 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 16:37:27.515304 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 16:37:27.515811 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050,"thread_start_us":1816,"threads_started":1}
I20260926 16:37:27.518317 14594 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41583
I20260926 16:37:27.518586 14594 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:27.519263 14594 heartbeater.cc:511] Master 127.10.110.62:41583 requested a full tablet report, sending...
I20260926 16:37:27.520669 14271 ts_manager.cc:194] Registered new tserver with Master: de04e6e896ef4032b99eddd02deed4ba (127.10.110.2:36793)
I20260926 16:37:27.521597 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:35919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:27.536564 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41583
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:27.849464 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:27.849756 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:27.849867 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:27.856976 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:27.857079 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:27.857187 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:27.867035 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:27.868284 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:27.869504 14599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:27.881884 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:27.881912 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:27.881942 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:27.882686 14599 server_base.cc:1034] running on GCE node
I20260926 16:37:27.883292 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:27.884238 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:27.885452 14599 hybrid_clock.cc:648] HybridClock initialized: now 1790440647885404 us; error 35 us; skew 500 ppm
I20260926 16:37:27.885792 14599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:27 dist-test-slave-whjp krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440647, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:27.888764 14599 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:27.890202 14599 webserver.cc:460] Webserver started at http://127.10.110.3:44047/ using document root <none> and password file <none>
I20260926 16:37:27.890867 14599 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:27.890980 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:27.891268 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:27.894240 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "29fa994232184080a6f4f47e7318ef02"
format_stamp: "Formatted at 2026-09-26 16:37:27 on dist-test-slave-whjp"
server_key: "d7bb340331e05b86ecf949fad0bef257"
server_key_iv: "6b2377c53721ae94b3e8ac9baaa4823d"
server_key_version: "encryptionkey@0"
I20260926 16:37:27.895004 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "29fa994232184080a6f4f47e7318ef02"
format_stamp: "Formatted at 2026-09-26 16:37:27 on dist-test-slave-whjp"
server_key: "d7bb340331e05b86ecf949fad0bef257"
server_key_iv: "6b2377c53721ae94b3e8ac9baaa4823d"
server_key_version: "encryptionkey@0"
I20260926 16:37:27.900322 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:37:27.903662 14614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.905030 14599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:27.905177 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "29fa994232184080a6f4f47e7318ef02"
format_stamp: "Formatted at 2026-09-26 16:37:27 on dist-test-slave-whjp"
server_key: "d7bb340331e05b86ecf949fad0bef257"
server_key_iv: "6b2377c53721ae94b3e8ac9baaa4823d"
server_key_version: "encryptionkey@0"
I20260926 16:37:27.905373 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:27.928004 14599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:27.932133 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:27.932399 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:27.933385 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:27.934824 14599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:27.934902 14599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.935009 14599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:27.935057 14599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:27.964881 14599 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:46197
I20260926 16:37:27.964918 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:46197 every 8 connection(s)
I20260926 16:37:27.966146 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:27.974103 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14599
I20260926 16:37:27.974303 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:27.974720 10680 external_mini_cluster.cc:1428] Setting key fd911e291bca71acc6d363d0fa94d87d
Sep 26 16:37:27 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440647, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:27.985757 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:27.968504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:57907 (local address 127.10.110.62:41583)
0926 16:37:27.968664 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:27.968668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:27.969822 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 16:37:27.971601 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:27.971607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:27.971685 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:27.971771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:27.973285 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.974126 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.975752 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.976377 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.976495 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:27.980369 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:27.980374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:27.980377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:27.980413 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:27.982796 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:27.983553 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:27.983559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:27.983564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:27.983696 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:27.984189 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:27.984193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:27.984195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:27.984477 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:27.984596 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:27.985335 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 16:37:27.985501 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:37:27.986732 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:27.968991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41583 (local address 127.10.110.3:57907)
0926 16:37:27.969648 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:27.969685 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:27.971336 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:27.972039 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:27.972047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:27.972559 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:27.973116 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:27.973138 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.974320 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.974325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:27.975617 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:27.975630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:27.976799 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:27.976807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:27.977120 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:27.977879 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:27.977904 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:27.980130 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:27.982941 (+ 2811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:27.982951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:27.982972 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:27.983429 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:27.983806 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:27.983812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:27.983850 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:27.984077 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:27.984610 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:27.984616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:27.984980 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 16:37:27.985254 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 16:37:27.985670 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":250,"threads_started":1}
I20260926 16:37:27.988029 14727 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41583
I20260926 16:37:27.988307 14727 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:27.989091 14727 heartbeater.cc:511] Master 127.10.110.62:41583 requested a full tablet report, sending...
I20260926 16:37:27.990896 14271 ts_manager.cc:194] Registered new tserver with Master: 29fa994232184080a6f4f47e7318ef02 (127.10.110.3:46197)
I20260926 16:37:27.991715 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:57907
I20260926 16:37:28.002812 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:28.049480 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:28.035387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.10.110.1:35155)
0926 16:37:28.035991 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:28.036000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:28.036030 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:37:28.036090 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:28.036095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:28.036326 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:28.036510 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:28.037868 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.038855 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.040094 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.040616 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.040733 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:28.044424 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:28.044435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:28.044448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:28.044477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:28.046930 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.047544 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.047552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.047554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.047680 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.048058 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.048062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.048064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.048377 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:28.048652 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:28.048928 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 16:37:28.049160 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260926 16:37:28.058046 14460 heartbeater.cc:503] Master 127.10.110.62:41583 was elected leader, sending a full tablet report...
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:28.065927 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:28.053211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.10.110.62:41583)
0926 16:37:28.053402 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:28.053408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:28.053601 (+ 193us) server_negotiation.cc:407] Connection header received
0926 16:37:28.053753 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:28.053758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:28.053820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:28.053895 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:28.055022 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.055729 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.056907 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.057291 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.057399 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:28.061176 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:28.061181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:28.061184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:28.061213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:28.063487 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.064118 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.064123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.064125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.064281 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.064601 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.064605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.064607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.064915 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:28.065137 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:28.065439 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:37:28.065642 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:37:28.066458 14271 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724
I20260926 16:37:28.067350 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14332
I20260926 16:37:28.079505 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:28.068119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.10.110.1:35155)
0926 16:37:28.068315 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:28.068320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:28.068506 (+ 186us) server_negotiation.cc:407] Connection header received
0926 16:37:28.068619 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:28.068623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:28.068693 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:28.068765 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:28.069804 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.070605 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.072138 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.072771 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.072896 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:28.075003 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:28.075008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:28.075011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:28.075041 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:28.077415 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.077957 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.077960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.077963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.078077 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.078334 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.078337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.078339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.078552 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:28.078726 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:28.079008 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:37:28.079212 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:28.080078 14455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:28.160458 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14332
I20260926 16:37:28.173453 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14466
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:28.188793 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:28.174316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.10.110.2:36793)
0926 16:37:28.174787 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:28.174795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:28.174820 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:37:28.174878 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:28.174884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:28.175094 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:28.175247 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:28.176520 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.177503 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.178725 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.179267 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.179388 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:28.183332 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:28.183342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:28.183357 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:28.183392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:28.186095 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.186732 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.186740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.186742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.186862 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.187207 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.187211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.187213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.187559 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:28.187877 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:28.188193 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 16:37:28.188460 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260926 16:37:28.190029 14589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:28.270680 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14466
I20260926 16:37:28.282893 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14599
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:28.298225 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:28.283942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.10.110.3:46197)
0926 16:37:28.284507 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:28.284513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:28.284541 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:37:28.284596 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:28.284602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:28.284834 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:28.284954 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:28.286150 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.287197 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.288515 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:28.289124 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:28.289236 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:28.293108 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:28.293119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:28.293133 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:28.293171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:28.295805 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.296474 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.296481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.296483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.296609 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:28.296921 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:28.296924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:28.296926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:28.297254 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:28.297474 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:28.297719 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 16:37:28.297916 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":277,"threads_started":1}
I20260926 16:37:28.299449 14722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:28.385315 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14599
I20260926 16:37:28.398126 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14239
I20260926 16:37:28.399302 14301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:28.500161 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14239
2026-09-26T16:37:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3600 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14746](info): set up 2 sockets
Sep 26 16:37:28 dist-test-slave-whjp krb5kdc[14746](info): commencing operation
krb5kdc: starting...
W20260926 16:37:29.505066 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:29 dist-test-slave-whjp krb5kdc[14746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:29.692368 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36783
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:36783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:29.999168 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:29.999468 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:29.999558 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:30.006897 14759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:30.007012 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:30.007057 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:30.007113 14759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:30.007159 14759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:30.017158 14759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:36783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:30.018471 14759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:30.019701 14759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:30.030359 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:30.030426 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:30.030404 14767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:30.031162 14759 server_base.cc:1034] running on GCE node
I20260926 16:37:30.031988 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:30.033298 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:30.034539 14759 hybrid_clock.cc:648] HybridClock initialized: now 1790440650034481 us; error 43 us; skew 500 ppm
I20260926 16:37:30.034899 14759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:30 dist-test-slave-whjp krb5kdc[14746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440650, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:30.041944 14759 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:30.043116 14759 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:30.043216 14759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:30.043530 14759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:30.046612 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "77c64fd70aba475289fddb10d5a42a5e"
format_stamp: "Formatted at 2026-09-26 16:37:30 on dist-test-slave-whjp"
server_key: "773c1d87d56f36179e0b4f5f19ee5869"
server_key_iv: "d6e1af346883d3464f8652ef8fa93769"
server_key_version: "encryptionkey@0"
I20260926 16:37:30.047370 14759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "77c64fd70aba475289fddb10d5a42a5e"
format_stamp: "Formatted at 2026-09-26 16:37:30 on dist-test-slave-whjp"
server_key: "773c1d87d56f36179e0b4f5f19ee5869"
server_key_iv: "d6e1af346883d3464f8652ef8fa93769"
server_key_version: "encryptionkey@0"
I20260926 16:37:30.052352 14759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:37:30.055519 14773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:30.056806 14759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:30.056960 14759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "77c64fd70aba475289fddb10d5a42a5e"
format_stamp: "Formatted at 2026-09-26 16:37:30 on dist-test-slave-whjp"
server_key: "773c1d87d56f36179e0b4f5f19ee5869"
server_key_iv: "d6e1af346883d3464f8652ef8fa93769"
server_key_version: "encryptionkey@0"
I20260926 16:37:30.057175 14759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:30.069831 14759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:30.073385 14759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:30.073612 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:30.090874 14759 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:36783
I20260926 16:37:30.090852 14824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:36783 every 8 connection(s)
I20260926 16:37:30.091771 14759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:30.095140 14825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:30.098554 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14759
I20260926 16:37:30.098783 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:30.099504 10680 external_mini_cluster.cc:1428] Setting key 5d1637adff451c3db421657533c47243
I20260926 16:37:30.108189 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e: Bootstrap starting.
I20260926 16:37:30.111986 14825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:30.113265 14825 log.cc:824] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:37:30 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:30.116443 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e: No bootstrap required, opened a new log
I20260926 16:37:30.122480 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:30.101522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.10.110.62:36783)
0926 16:37:30.102269 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:30.102280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:30.102319 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:37:30.103850 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:30.103879 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:30.104304 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:30.104632 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:30.106641 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:30.108006 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:30.109715 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:30.110477 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:30.110630 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:30.115436 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:30.115444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:30.115461 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:30.115499 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:30.118441 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:30.119273 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:30.119280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:30.119289 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:30.119454 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:30.119820 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:30.119823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:30.119825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:30.120393 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:30.120602 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:30.121073 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 16:37:30.121442 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":357,"threads_started":1}
I20260926 16:37:30.125149 14825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c64fd70aba475289fddb10d5a42a5e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36783 } }
I20260926 16:37:30.125582 14825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:30.125659 14825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c64fd70aba475289fddb10d5a42a5e, State: Initialized, Role: FOLLOWER
I20260926 16:37:30.126130 14825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c64fd70aba475289fddb10d5a42a5e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36783 } }
I20260926 16:37:30.126327 14825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:30.126420 14825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:30.126536 14825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:30.129693 14825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c64fd70aba475289fddb10d5a42a5e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36783 } }
I20260926 16:37:30.130276 14825 leader_election.cc:302] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c64fd70aba475289fddb10d5a42a5e; no voters:
I20260926 16:37:30.130985 14825 leader_election.cc:288] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:30.131088 14830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:30.132087 14830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [term 1 LEADER]: Becoming Leader. State: Replica: 77c64fd70aba475289fddb10d5a42a5e, State: Running, Role: LEADER
I20260926 16:37:30.132603 14830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c64fd70aba475289fddb10d5a42a5e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36783 } }
I20260926 16:37:30.133050 14825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:30.135118 14832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c64fd70aba475289fddb10d5a42a5e. Latest consensus state: current_term: 1 leader_uuid: "77c64fd70aba475289fddb10d5a42a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c64fd70aba475289fddb10d5a42a5e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36783 } } }
I20260926 16:37:30.135478 14832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:30.136132 14831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c64fd70aba475289fddb10d5a42a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c64fd70aba475289fddb10d5a42a5e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36783 } } }
I20260926 16:37:30.136432 14831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:30.136495 14835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:30.141994 14835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:30.150229 14835 catalog_manager.cc:1347] Generated new cluster ID: 23f13af3cd7646338afe36ac02684add
I20260926 16:37:30.150341 14835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:30.173307 14835 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:30.174619 14835 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:30.183885 14835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e: Generated new TSK 0
I20260926 16:37:30.184723 14835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:30.234805 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36783
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:30.547515 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:30.547775 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:30.547917 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:30.555357 14851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:30.555467 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:30.555629 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:30.565387 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:30.566665 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:30.567976 14851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:30.580047 14856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:30.579977 14857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:30.580008 14859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:30.581094 14851 server_base.cc:1034] running on GCE node
I20260926 16:37:30.581668 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:30.582557 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:30.583822 14851 hybrid_clock.cc:648] HybridClock initialized: now 1790440650583768 us; error 43 us; skew 500 ppm
I20260926 16:37:30.584192 14851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:30 dist-test-slave-whjp krb5kdc[14746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440650, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:30.598717 14851 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:30.600032 14851 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:30.600143 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:30.600445 14851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:30.603324 14851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "4bf3df2b73d04c4195b631062b87f62d"
format_stamp: "Formatted at 2026-09-26 16:37:30 on dist-test-slave-whjp"
server_key: "6717af2d72d8f1f8ffb9c90473a7bd70"
server_key_iv: "4e4a8b3ce1f4eac7cbef7ea96f0cacff"
server_key_version: "encryptionkey@0"
I20260926 16:37:30.604135 14851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "4bf3df2b73d04c4195b631062b87f62d"
format_stamp: "Formatted at 2026-09-26 16:37:30 on dist-test-slave-whjp"
server_key: "6717af2d72d8f1f8ffb9c90473a7bd70"
server_key_iv: "4e4a8b3ce1f4eac7cbef7ea96f0cacff"
server_key_version: "encryptionkey@0"
I20260926 16:37:30.609047 14851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:37:30.612440 14865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:30.613685 14851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:30.613843 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "4bf3df2b73d04c4195b631062b87f62d"
format_stamp: "Formatted at 2026-09-26 16:37:30 on dist-test-slave-whjp"
server_key: "6717af2d72d8f1f8ffb9c90473a7bd70"
server_key_iv: "4e4a8b3ce1f4eac7cbef7ea96f0cacff"
server_key_version: "encryptionkey@0"
I20260926 16:37:30.614037 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:30.627734 14851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:30.632169 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:30.632426 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:30.633476 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:30.634948 14851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:30.635042 14851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:30.635146 14851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:30.635213 14851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:30.664520 14851 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:45859
I20260926 16:37:30.664558 14977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:45859 every 8 connection(s)
I20260926 16:37:30.665314 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:30.668708 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14851
I20260926 16:37:30.668910 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:30.669561 10680 external_mini_cluster.cc:1428] Setting key 4d3d850758f2dbd2d593e32e598d975a
Sep 26 16:37:30 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440650, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:30.683888 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:30.667495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:39837 (local address 127.10.110.62:36783)
0926 16:37:30.667916 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:30.667920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:30.668811 (+ 891us) server_negotiation.cc:407] Connection header received
0926 16:37:30.670718 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:30.670723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:30.670806 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:30.670896 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:30.672477 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:30.673173 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:30.674655 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:30.675075 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:30.675181 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:30.678628 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:30.678632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:30.678636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:30.678667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:30.680950 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:30.681648 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:30.681652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:30.681654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:30.681769 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:30.682164 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:30.682167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:30.682169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:30.682438 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:30.682563 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:30.683264 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 16:37:30.683477 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":198,"threads_started":1}
I20260926 16:37:30.684881 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:30.667994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36783 (local address 127.10.110.1:39837)
0926 16:37:30.668652 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:30.668702 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:30.670458 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:30.671158 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:30.671169 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:30.671726 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:30.672322 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:30.672341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:30.673306 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:30.673311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:30.674525 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:30.674536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:30.675295 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:30.675303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:30.675504 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:30.676150 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:30.676177 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:30.678410 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:30.681084 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:30.681094 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:30.681113 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:30.681527 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:30.681866 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:30.681871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:30.681875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:30.682056 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:30.682554 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:30.682562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:30.682906 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 16:37:30.683143 (+ 237us) client_negotiation.cc:240] Negotiation successful
0926 16:37:30.683662 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":285,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:37:30.686363 14978 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36783
I20260926 16:37:30.686638 14978 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:30.687273 14978 heartbeater.cc:511] Master 127.10.110.62:36783 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:37:30.689323 14790 ts_manager.cc:194] Registered new tserver with Master: 4bf3df2b73d04c4195b631062b87f62d (127.10.110.1:45859)
I20260926 16:37:30.690716 14790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:39837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:30.710671 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36783
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:31.030778 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:31.031051 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:31.031144 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:31.038395 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:31.038511 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:31.038635 14984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:31.048310 14984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:31.049567 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:31.050786 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:31.062860 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:31.062894 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:31.062912 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:31.063606 14984 server_base.cc:1034] running on GCE node
I20260926 16:37:31.064205 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:31.065099 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:31.066274 14984 hybrid_clock.cc:648] HybridClock initialized: now 1790440651066228 us; error 41 us; skew 500 ppm
I20260926 16:37:31.066617 14984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:31 dist-test-slave-whjp krb5kdc[14746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440651, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:31.069963 14984 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:31.071029 14984 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:31.071149 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:31.071442 14984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:31.074294 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "3182403ed43f4db1be3ebbd1f28b0282"
format_stamp: "Formatted at 2026-09-26 16:37:31 on dist-test-slave-whjp"
server_key: "d0541f2fb748915bd22bf02b46047867"
server_key_iv: "89ff77d08091d13cebadea029fac2654"
server_key_version: "encryptionkey@0"
I20260926 16:37:31.075054 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "3182403ed43f4db1be3ebbd1f28b0282"
format_stamp: "Formatted at 2026-09-26 16:37:31 on dist-test-slave-whjp"
server_key: "d0541f2fb748915bd22bf02b46047867"
server_key_iv: "89ff77d08091d13cebadea029fac2654"
server_key_version: "encryptionkey@0"
I20260926 16:37:31.081135 14984 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260926 16:37:31.085183 14998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:31.086555 14984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:31.086692 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "3182403ed43f4db1be3ebbd1f28b0282"
format_stamp: "Formatted at 2026-09-26 16:37:31 on dist-test-slave-whjp"
server_key: "d0541f2fb748915bd22bf02b46047867"
server_key_iv: "89ff77d08091d13cebadea029fac2654"
server_key_version: "encryptionkey@0"
I20260926 16:37:31.086889 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:31.107517 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:31.111405 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:31.111629 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:31.112594 14984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:31.113866 14984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:31.113948 14984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:31.114049 14984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:31.114135 14984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:31.143152 14984 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:37575
I20260926 16:37:31.143164 15110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:37575 every 8 connection(s)
I20260926 16:37:31.144002 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:31.148085 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 14984
I20260926 16:37:31.148247 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:31.148665 10680 external_mini_cluster.cc:1428] Setting key fa7e35059d62bb71f801da016c2e524d
Sep 26 16:37:31 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440651, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:31.163096 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.146259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:38047 (local address 127.10.110.62:36783)
0926 16:37:31.146425 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.146430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.147561 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 16:37:31.149382 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.149387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.149471 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.149555 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:31.151005 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.151712 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.153228 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.153607 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.153718 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.157514 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:31.157519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:31.157522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:31.157554 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:31.159951 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:31.160866 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:31.160872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:31.160875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:31.161004 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:31.161467 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:31.161471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:31.161473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:31.161755 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:31.161864 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.162700 (+ 836us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.162863 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:37:31.164601 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.146756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36783 (local address 127.10.110.2:38047)
0926 16:37:31.147403 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.147440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.149119 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.149792 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.149813 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.150309 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:31.150849 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.150869 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.151883 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.151888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.153096 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.153108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.153805 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.153812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.154017 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.154637 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:31.154668 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:31.157296 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:31.160079 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:31.160088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:31.160105 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:31.160758 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:31.161099 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:31.161104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:31.161107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:31.161347 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:31.161872 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:31.161880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:31.162325 (+ 445us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.162633 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.163182 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:37:31.166020 15111 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36783
I20260926 16:37:31.166328 15111 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:31.166949 15111 heartbeater.cc:511] Master 127.10.110.62:36783 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:37:31.168413 14790 ts_manager.cc:194] Registered new tserver with Master: 3182403ed43f4db1be3ebbd1f28b0282 (127.10.110.2:37575)
I20260926 16:37:31.169173 14790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:38047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:31.192262 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36783
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:31.503305 15116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:31.503592 15116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:31.503726 15116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:31.511058 15116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:31.511193 15116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:31.511358 15116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:31.521174 15116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:31.522466 15116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:31.523746 15116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:31.536180 15122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:31.536286 15121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:31.536391 15124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:31.537089 15116 server_base.cc:1034] running on GCE node
I20260926 16:37:31.537704 15116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:31.538653 15116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:31.539875 15116 hybrid_clock.cc:648] HybridClock initialized: now 1790440651539792 us; error 45 us; skew 500 ppm
I20260926 16:37:31.540185 15116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:31 dist-test-slave-whjp krb5kdc[14746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440651, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:31.543068 15116 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:31.544163 15116 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:31.544255 15116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:31.544541 15116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:31.547354 15116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "e1657f659d6f41d297af11d606cb27e8"
format_stamp: "Formatted at 2026-09-26 16:37:31 on dist-test-slave-whjp"
server_key: "549e4d605465c8829f47139b00e7a82c"
server_key_iv: "99ce20e215ed76567a61733cdda58427"
server_key_version: "encryptionkey@0"
I20260926 16:37:31.548162 15116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "e1657f659d6f41d297af11d606cb27e8"
format_stamp: "Formatted at 2026-09-26 16:37:31 on dist-test-slave-whjp"
server_key: "549e4d605465c8829f47139b00e7a82c"
server_key_iv: "99ce20e215ed76567a61733cdda58427"
server_key_version: "encryptionkey@0"
I20260926 16:37:31.552970 15116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:37:31.556355 15130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:31.557624 15116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:31.557783 15116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "e1657f659d6f41d297af11d606cb27e8"
format_stamp: "Formatted at 2026-09-26 16:37:31 on dist-test-slave-whjp"
server_key: "549e4d605465c8829f47139b00e7a82c"
server_key_iv: "99ce20e215ed76567a61733cdda58427"
server_key_version: "encryptionkey@0"
I20260926 16:37:31.557973 15116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:31.604075 15116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:31.608134 15116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:31.608362 15116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:31.609292 15116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:31.610563 15116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:31.610647 15116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:31.610725 15116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:31.610782 15116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:31.640075 15116 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:40941
I20260926 16:37:31.640139 15242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:40941 every 8 connection(s)
I20260926 16:37:31.640869 15116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:31.649852 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15116
I20260926 16:37:31.650041 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:31.650430 10680 external_mini_cluster.cc:1428] Setting key 7eb4674a7e4fe2a8b56d39b12acd8206
Sep 26 16:37:31 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440651, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:31.659045 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.643024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:39301 (local address 127.10.110.62:36783)
0926 16:37:31.643238 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.643244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.644439 (+ 1195us) server_negotiation.cc:407] Connection header received
0926 16:37:31.646162 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.646166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.646267 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.646352 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:31.647813 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.648564 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.650150 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.650547 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.650660 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.654199 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:31.654223 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:31.654226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:31.654257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:31.656438 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:31.657055 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:31.657058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:31.657061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:31.657166 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:31.657490 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:31.657493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:31.657495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:31.657746 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:31.657849 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.658507 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.658719 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:37:31.660000 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.643534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36783 (local address 127.10.110.3:39301)
0926 16:37:31.644267 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.644303 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.645904 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.646580 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.646590 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.647091 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:31.647648 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.647670 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.648728 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.648732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.650005 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.650020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.650761 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.650768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.650987 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.651690 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:31.651719 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:31.654007 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:31.656554 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:31.656560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:31.656574 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:31.656969 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:31.657258 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:31.657261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:31.657263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:31.657418 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:31.657832 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:31.657837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:31.658141 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.658405 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.658899 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":289,"threads_started":1}
I20260926 16:37:31.661291 15243 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36783
I20260926 16:37:31.661571 15243 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:31.662233 15243 heartbeater.cc:511] Master 127.10.110.62:36783 requested a full tablet report, sending...
I20260926 16:37:31.663527 14790 ts_manager.cc:194] Registered new tserver with Master: e1657f659d6f41d297af11d606cb27e8 (127.10.110.3:40941)
I20260926 16:37:31.664599 14790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:39301
I20260926 16:37:31.665879 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:37:31.682260 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.670972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.10.110.62:36783)
0926 16:37:31.671184 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.671191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.671490 (+ 299us) server_negotiation.cc:407] Connection header received
0926 16:37:31.671631 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.671634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.671692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.671779 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:31.673079 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.673787 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.674949 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.675388 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.675482 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.677761 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:31.677766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:31.677770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:31.677801 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:31.680151 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:31.680706 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:31.680709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:31.680711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:31.680828 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:31.681092 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:31.681095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:31.681097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:31.681390 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:31.681562 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.681818 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.681974 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:37:31.686184 14789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:31.692894 14789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:31.693975 14978 heartbeater.cc:503] Master 127.10.110.62:36783 was elected leader, sending a full tablet report...
I20260926 16:37:31.716343 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.706173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.10.110.2:37575)
0926 16:37:31.708956 (+ 2783us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.708962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.708983 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:37:31.709055 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.709060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.709260 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.709437 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:31.710095 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.711601 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.714167 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.715311 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.715429 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.715616 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.715897 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.716056 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2667,"thread_start_us":2583,"threads_started":1}
I20260926 16:37:31.716351 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.705155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37575 (local address 127.0.0.1:42740)
0926 16:37:31.708254 (+ 3099us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.708270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.708370 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.709453 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.709460 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.709490 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:31.709889 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.709901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.711734 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.711739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.713992 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.714006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.715499 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.715505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.715676 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.715720 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.715822 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.716022 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2965,"thread_start_us":5769,"threads_started":1}
I20260926 16:37:31.717155 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.704966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:40941 (local address 127.0.0.1:47500)
0926 16:37:31.706206 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.706242 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.706414 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.710565 (+ 4151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.710569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.710591 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:31.711025 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.711042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.712750 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.712753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.714684 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.714697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.716464 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.716470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.716624 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.716656 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.716718 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.716839 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":3026,"threads_started":1}
I20260926 16:37:31.718353 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.705699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.10.110.3:40941)
0926 16:37:31.710060 (+ 4361us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.710067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.710100 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:37:31.710186 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.710193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.710407 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.710559 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:31.711249 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.712629 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.714850 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.716258 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.717515 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.717790 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.717855 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.718014 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4210,"thread_start_us":4037,"threads_started":1}
I20260926 16:37:31.718737 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.705499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:45859 (local address 127.0.0.1:37624)
0926 16:37:31.711228 (+ 5729us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.711241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.711336 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.711926 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.711930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.711955 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:31.712358 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.712370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.714196 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.714201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.716034 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.716048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.718051 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.718057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.718228 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.718257 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.718330 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.718444 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5606,"thread_start_us":963,"threads_started":1}
I20260926 16:37:31.719437 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.706246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.10.110.1:45859)
0926 16:37:31.708668 (+ 2422us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.708675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.711329 (+ 2654us) server_negotiation.cc:407] Connection header received
0926 16:37:31.711457 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.711462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.711614 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.711766 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:31.712538 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.714070 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.716212 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.717588 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.717690 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.717878 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.718977 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.719135 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2246,"thread_start_us":2095,"threads_started":1}
I20260926 16:37:31.720302 15046 tablet_service.cc:1467] Processing CreateTablet for tablet a5fac230be894eefa8f60bc8f1ac4fa8 (DEFAULT_TABLE table=test-table [id=8fd9c061a6ff4394b833a148f652fb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:31.721765 15046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fac230be894eefa8f60bc8f1ac4fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:31.721776 15178 tablet_service.cc:1467] Processing CreateTablet for tablet a5fac230be894eefa8f60bc8f1ac4fa8 (DEFAULT_TABLE table=test-table [id=8fd9c061a6ff4394b833a148f652fb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:31.722931 15178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fac230be894eefa8f60bc8f1ac4fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:31.722723 14913 tablet_service.cc:1467] Processing CreateTablet for tablet a5fac230be894eefa8f60bc8f1ac4fa8 (DEFAULT_TABLE table=test-table [id=8fd9c061a6ff4394b833a148f652fb3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:31.724309 14913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5fac230be894eefa8f60bc8f1ac4fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:31.733425 15261 tablet_bootstrap.cc:492] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Bootstrap starting.
I20260926 16:37:31.735653 15262 tablet_bootstrap.cc:492] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Bootstrap starting.
I20260926 16:37:31.737084 15261 tablet_bootstrap.cc:654] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:31.738358 15261 log.cc:824] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:31.738898 15263 tablet_bootstrap.cc:492] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Bootstrap starting.
I20260926 16:37:31.739223 15262 tablet_bootstrap.cc:654] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:31.740844 15262 log.cc:824] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:31.741742 15263 tablet_bootstrap.cc:654] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:31.742713 15263 log.cc:824] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:31.744248 15262 tablet_bootstrap.cc:492] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: No bootstrap required, opened a new log
I20260926 16:37:31.744493 15262 ts_tablet_manager.cc:1397] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 16:37:31.745260 15263 tablet_bootstrap.cc:492] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: No bootstrap required, opened a new log
I20260926 16:37:31.745549 15263 ts_tablet_manager.cc:1397] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260926 16:37:31.746451 15261 tablet_bootstrap.cc:492] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: No bootstrap required, opened a new log
I20260926 16:37:31.746718 15261 ts_tablet_manager.cc:1397] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260926 16:37:31.752251 15263 raft_consensus.cc:348] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.752584 15263 raft_consensus.cc:374] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:31.752687 15263 raft_consensus.cc:729] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf3df2b73d04c4195b631062b87f62d, State: Initialized, Role: FOLLOWER
I20260926 16:37:31.753239 15263 consensus_queue.cc:260] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.754681 15263 ts_tablet_manager.cc:1428] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260926 16:37:31.755249 15262 raft_consensus.cc:348] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.755635 15262 raft_consensus.cc:374] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:31.755725 15262 raft_consensus.cc:729] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3182403ed43f4db1be3ebbd1f28b0282, State: Initialized, Role: FOLLOWER
I20260926 16:37:31.756424 15262 consensus_queue.cc:260] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.756412 15261 raft_consensus.cc:348] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.756801 15261 raft_consensus.cc:374] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:31.756902 15261 raft_consensus.cc:729] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1657f659d6f41d297af11d606cb27e8, State: Initialized, Role: FOLLOWER
I20260926 16:37:31.757468 15261 consensus_queue.cc:260] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.758008 15111 heartbeater.cc:503] Master 127.10.110.62:36783 was elected leader, sending a full tablet report...
I20260926 16:37:31.758553 15262 ts_tablet_manager.cc:1428] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 16:37:31.759891 15243 heartbeater.cc:503] Master 127.10.110.62:36783 was elected leader, sending a full tablet report...
I20260926 16:37:31.760588 15261 ts_tablet_manager.cc:1428] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260926 16:37:31.875921 15267 raft_consensus.cc:482] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:31.876230 15267 raft_consensus.cc:504] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.877851 15267 leader_election.cc:288] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1657f659d6f41d297af11d606cb27e8 (127.10.110.3:40941), 3182403ed43f4db1be3ebbd1f28b0282 (127.10.110.2:37575)
I20260926 16:37:31.883805 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.878335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:50365 (local address 127.10.110.2:37575)
0926 16:37:31.878623 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.878629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.878929 (+ 300us) server_negotiation.cc:407] Connection header received
0926 16:37:31.879118 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.879122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.879189 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.879271 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:31.879763 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.880496 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.882194 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.882943 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.883085 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.883159 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.883441 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.883561 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260926 16:37:31.883873 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.878133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:37575 (local address 127.10.110.1:50365)
0926 16:37:31.878821 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.878839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.878992 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.879283 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.879288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.879310 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:31.879622 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.879632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.880615 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.880620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.882050 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.882062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.883123 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.883129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.883301 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.883324 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.883400 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.883518 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":399,"threads_started":1}
I20260926 16:37:31.883908 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.878200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:36345 (local address 127.10.110.3:40941)
0926 16:37:31.878433 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.878439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.879185 (+ 746us) server_negotiation.cc:407] Connection header received
0926 16:37:31.879350 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.879353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.879409 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.879478 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:31.880008 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.880688 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.882312 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.883058 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.883200 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.883268 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.883523 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.883638 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:37:31.883946 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.878225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:40941 (local address 127.10.110.1:36345)
0926 16:37:31.879089 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:31.879102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:31.879215 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:31.879520 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:31.879524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:31.879546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:31.879872 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.879884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.880829 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.880832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.882169 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:31.882181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.883276 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.883283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:31.883394 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.883415 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:37:31.883488 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 16:37:31.883600 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":564,"threads_started":1}
I20260926 16:37:31.884828 15066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fac230be894eefa8f60bc8f1ac4fa8" candidate_uuid: "4bf3df2b73d04c4195b631062b87f62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3182403ed43f4db1be3ebbd1f28b0282" is_pre_election: true
I20260926 16:37:31.884826 15198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fac230be894eefa8f60bc8f1ac4fa8" candidate_uuid: "4bf3df2b73d04c4195b631062b87f62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1657f659d6f41d297af11d606cb27e8" is_pre_election: true
I20260926 16:37:31.885272 15066 raft_consensus.cc:2454] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf3df2b73d04c4195b631062b87f62d in term 0.
I20260926 16:37:31.885269 15198 raft_consensus.cc:2454] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf3df2b73d04c4195b631062b87f62d in term 0.
I20260926 16:37:31.885866 14866 leader_election.cc:302] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3182403ed43f4db1be3ebbd1f28b0282, 4bf3df2b73d04c4195b631062b87f62d, e1657f659d6f41d297af11d606cb27e8; no voters:
I20260926 16:37:31.886229 15267 raft_consensus.cc:2781] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:31.886353 15267 raft_consensus.cc:482] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:31.886445 15267 raft_consensus.cc:3037] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:31.888559 15267 raft_consensus.cc:504] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
I20260926 16:37:31.889128 15267 leader_election.cc:288] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [CANDIDATE]: Term 1 election: Requested vote from peers e1657f659d6f41d297af11d606cb27e8 (127.10.110.3:40941), 3182403ed43f4db1be3ebbd1f28b0282 (127.10.110.2:37575)
I20260926 16:37:31.889510 15198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fac230be894eefa8f60bc8f1ac4fa8" candidate_uuid: "4bf3df2b73d04c4195b631062b87f62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1657f659d6f41d297af11d606cb27e8"
I20260926 16:37:31.889599 15066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5fac230be894eefa8f60bc8f1ac4fa8" candidate_uuid: "4bf3df2b73d04c4195b631062b87f62d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3182403ed43f4db1be3ebbd1f28b0282"
I20260926 16:37:31.889727 15198 raft_consensus.cc:3037] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:31.889824 15066 raft_consensus.cc:3037] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:31.892064 15198 raft_consensus.cc:2454] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf3df2b73d04c4195b631062b87f62d in term 1.
I20260926 16:37:31.892102 15066 raft_consensus.cc:2454] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf3df2b73d04c4195b631062b87f62d in term 1.
W20260926 16:37:31.892223 15244 tablet.cc:2369] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:31.892428 14866 leader_election.cc:302] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3182403ed43f4db1be3ebbd1f28b0282, 4bf3df2b73d04c4195b631062b87f62d; no voters:
I20260926 16:37:31.892686 15267 raft_consensus.cc:2781] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:31.893105 15267 raft_consensus.cc:686] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 1 LEADER]: Becoming Leader. State: Replica: 4bf3df2b73d04c4195b631062b87f62d, State: Running, Role: LEADER
I20260926 16:37:31.893576 15267 consensus_queue.cc:237] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } }
W20260926 16:37:31.895634 15112 tablet.cc:2369] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:31.897971 14789 catalog_manager.cc:5495] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bf3df2b73d04c4195b631062b87f62d (127.10.110.1). New cstate: current_term: 1 leader_uuid: "4bf3df2b73d04c4195b631062b87f62d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bf3df2b73d04c4195b631062b87f62d" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45859 } health_report { overall_health: HEALTHY } } }
W20260926 16:37:31.917461 14979 tablet.cc:2369] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:31.953991 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.949151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.10.110.1:45859)
0926 16:37:31.949309 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.949314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:31.949422 (+ 108us) server_negotiation.cc:407] Connection header received
0926 16:37:31.949610 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:31.949614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:31.949706 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:31.949839 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:31.950194 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.950883 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.952231 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:31.952618 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:31.952715 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:31.953084 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:31.953562 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:31.953669 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:31.953736 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 16:37:31.953797 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:37:31.962266 15198 raft_consensus.cc:1261] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 1 FOLLOWER]: Refusing update from remote peer 4bf3df2b73d04c4195b631062b87f62d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:31.962266 15066 raft_consensus.cc:1261] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 1 FOLLOWER]: Refusing update from remote peer 4bf3df2b73d04c4195b631062b87f62d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:31.963246 15272 consensus_queue.cc:1035] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1657f659d6f41d297af11d606cb27e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:31.963563 15267 consensus_queue.cc:1035] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3182403ed43f4db1be3ebbd1f28b0282" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:31.975080 15275 mvcc.cc:205] Tried to move back new op lower bound from 7333644910426394624 to 7333644910159486976. Current Snapshot: MvccSnapshot[applied={T|T < 7333644910426394624}]
I20260926 16:37:31.987411 14789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:31.987700 14789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:31.991587 14789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c64fd70aba475289fddb10d5a42a5e: Sending DeleteTablet for 3 replicas of tablet a5fac230be894eefa8f60bc8f1ac4fa8
I20260926 16:37:31.992929 15178 tablet_service.cc:1514] Processing DeleteTablet for tablet a5fac230be894eefa8f60bc8f1ac4fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:31 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:47500
I20260926 16:37:31.992929 15046 tablet_service.cc:1514] Processing DeleteTablet for tablet a5fac230be894eefa8f60bc8f1ac4fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:31 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42740
I20260926 16:37:31.993484 14913 tablet_service.cc:1514] Processing DeleteTablet for tablet a5fac230be894eefa8f60bc8f1ac4fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:31 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:37624
I20260926 16:37:31.995566 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14851
I20260926 16:37:32.005378 15285 tablet_replica.cc:330] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: stopping tablet replica
I20260926 16:37:32.006078 15285 raft_consensus.cc:2229] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:32.006758 15285 raft_consensus.cc:2258] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:32.008267 15284 tablet_replica.cc:330] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: stopping tablet replica
I20260926 16:37:32.008265 15286 tablet_replica.cc:330] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: stopping tablet replica
I20260926 16:37:32.008699 15286 raft_consensus.cc:2229] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:32.008805 15284 raft_consensus.cc:2229] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:32.009014 15285 ts_tablet_manager.cc:1905] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:32.009377 15284 raft_consensus.cc:2258] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:32.009505 15286 raft_consensus.cc:2258] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:32.011737 15284 ts_tablet_manager.cc:1905] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:32.013446 15286 ts_tablet_manager.cc:1905] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:32.015822 15285 ts_tablet_manager.cc:1918] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:32.016072 15285 log.cc:1196] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/a5fac230be894eefa8f60bc8f1ac4fa8
I20260926 16:37:32.016647 15285 ts_tablet_manager.cc:1939] T a5fac230be894eefa8f60bc8f1ac4fa8 P 3182403ed43f4db1be3ebbd1f28b0282: Deleting consensus metadata
I20260926 16:37:32.018517 14774 catalog_manager.cc:4841] TS 3182403ed43f4db1be3ebbd1f28b0282 (127.10.110.2:37575): tablet a5fac230be894eefa8f60bc8f1ac4fa8 (table test-table [id=8fd9c061a6ff4394b833a148f652fb3b]) successfully deleted
Sep 26 16:37:32 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:32.023161 15284 ts_tablet_manager.cc:1918] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:32.023542 15286 ts_tablet_manager.cc:1918] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:32.023615 15284 log.cc:1196] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/a5fac230be894eefa8f60bc8f1ac4fa8
I20260926 16:37:32.023854 15286 log.cc:1196] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/a5fac230be894eefa8f60bc8f1ac4fa8
I20260926 16:37:32.024152 15284 ts_tablet_manager.cc:1939] T a5fac230be894eefa8f60bc8f1ac4fa8 P e1657f659d6f41d297af11d606cb27e8: Deleting consensus metadata
I20260926 16:37:32.024407 15286 ts_tablet_manager.cc:1939] T a5fac230be894eefa8f60bc8f1ac4fa8 P 4bf3df2b73d04c4195b631062b87f62d: Deleting consensus metadata
I20260926 16:37:32.026058 14777 catalog_manager.cc:4841] TS 4bf3df2b73d04c4195b631062b87f62d (127.10.110.1:45859): tablet a5fac230be894eefa8f60bc8f1ac4fa8 (table test-table [id=8fd9c061a6ff4394b833a148f652fb3b]) successfully deleted
I20260926 16:37:32.026173 14776 catalog_manager.cc:4841] TS e1657f659d6f41d297af11d606cb27e8 (127.10.110.3:40941): tablet a5fac230be894eefa8f60bc8f1ac4fa8 (table test-table [id=8fd9c061a6ff4394b833a148f652fb3b]) successfully deleted
I20260926 16:37:32.029774 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:31.996733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.10.110.1:45859)
0926 16:37:31.996919 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:31.996925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:32.010371 (+ 13446us) server_negotiation.cc:407] Connection header received
0926 16:37:32.010638 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:32.010643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:32.010736 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:32.010811 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:32.012656 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:32.013779 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:32.015628 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:32.016327 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:32.016428 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:32.021894 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:32.021912 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:32.021935 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:32.021992 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:32.026635 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:32.027444 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:32.027454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:32.027458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:32.027633 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:32.028064 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:32.028070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:32.028073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:32.028565 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:32.028860 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:32.029117 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 16:37:32.029374 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:37:32.030474 14973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:32.152612 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14851
I20260926 16:37:32.172744 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14984
Sep 26 16:37:32 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:32.186805 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:32.173494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.10.110.2:37575)
0926 16:37:32.173679 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:32.173683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:32.173701 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:37:32.173789 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:32.173792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:32.173856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:32.173934 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:32.175089 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:32.175919 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:32.177104 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:32.177502 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:32.177600 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:32.181373 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:32.181388 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:32.181404 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:32.181442 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:32.184222 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:32.184954 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:32.184963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:32.184965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:32.185094 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:32.185389 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:32.185392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:32.185394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:32.185815 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:32.186033 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:32.186294 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 16:37:32.186460 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:37:32.187506 15106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:32.273931 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14984
I20260926 16:37:32.286832 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15116
Sep 26 16:37:32 dist-test-slave-whjp krb5kdc[14746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:32.300506 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:32.287539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.10.110.3:40941)
0926 16:37:32.287713 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:32.287719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:32.287739 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:37:32.287964 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:32.287970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:32.288038 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:32.288116 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:32.289320 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:32.290038 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:32.291225 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:32.291597 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:32.291702 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:32.295555 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:32.295569 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:32.295585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:32.295620 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:32.298047 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:32.298622 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:32.298629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:32.298631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:32.298761 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:32.299091 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:32.299095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:32.299097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:32.299418 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:32.299697 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:32.299997 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 16:37:32.300222 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:37:32.301095 15238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:32.389432 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15116
I20260926 16:37:32.402288 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 14759
I20260926 16:37:32.403331 14820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:32.508394 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 14759
2026-09-26T16:37:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4008 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T16:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:32Z Disabled control of system clock
I20260926 16:37:32.567215 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34831
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:37:32.896330 15295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:32.896620 15295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:32.896744 15295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:32.904011 15295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:32.904124 15295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:32.904167 15295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:32.904242 15295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:32.904295 15295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:32.914198 15295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:32.915504 15295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:32.917375 15295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:32.928439 15303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:32.928463 15301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:32.928398 15300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:32.929466 15295 server_base.cc:1034] running on GCE node
I20260926 16:37:32.930227 15295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:32.931589 15295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:32.932777 15295 hybrid_clock.cc:648] HybridClock initialized: now 1790440652932724 us; error 47 us; skew 500 ppm
I20260926 16:37:32.933100 15295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:32.935261 15295 webserver.cc:460] Webserver started at http://127.10.110.62:43385/ using document root <none> and password file <none>
I20260926 16:37:32.935987 15295 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:32.936077 15295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:32.936378 15295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:32.939138 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "fef783cf73fd40ebab052ba927b31260"
format_stamp: "Formatted at 2026-09-26 16:37:32 on dist-test-slave-whjp"
server_key: "2fca497db951704c614c67500b766561"
server_key_iv: "cfb4daa5d91578d8271c982c4c603aa9"
server_key_version: "encryptionkey@0"
I20260926 16:37:32.939915 15295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "fef783cf73fd40ebab052ba927b31260"
format_stamp: "Formatted at 2026-09-26 16:37:32 on dist-test-slave-whjp"
server_key: "2fca497db951704c614c67500b766561"
server_key_iv: "cfb4daa5d91578d8271c982c4c603aa9"
server_key_version: "encryptionkey@0"
I20260926 16:37:32.945001 15295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:37:32.948338 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:32.949661 15295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:37:32.949843 15295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "fef783cf73fd40ebab052ba927b31260"
format_stamp: "Formatted at 2026-09-26 16:37:32 on dist-test-slave-whjp"
server_key: "2fca497db951704c614c67500b766561"
server_key_iv: "cfb4daa5d91578d8271c982c4c603aa9"
server_key_version: "encryptionkey@0"
I20260926 16:37:32.950031 15295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:32.955538 15295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:32.956333 15295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:32.956595 15295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:32.973080 15295 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:34831
I20260926 16:37:32.973173 15360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:34831 every 8 connection(s)
I20260926 16:37:32.974413 15295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:32.974790 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15295
I20260926 16:37:32.974962 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:32.975816 10680 external_mini_cluster.cc:1428] Setting key 05e06357937b5a664b664d7a215c4f4b
I20260926 16:37:32.979910 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:32.983652 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:32.977720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.10.110.62:34831)
0926 16:37:32.978641 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:32.978653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:32.978693 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:37:32.980313 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:32.980336 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:32.980731 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:32.981060 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:32.982065 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:32.982072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:32.982084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:32.982113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:32.982179 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:32.982195 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:32.982415 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:32.982555 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 16:37:32.982790 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":281,"threads_started":1}
I20260926 16:37:32.990674 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260: Bootstrap starting.
I20260926 16:37:32.993876 15361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:32.994856 15361 log.cc:824] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:32.997756 15361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260: No bootstrap required, opened a new log
I20260926 16:37:33.005544 15361 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef783cf73fd40ebab052ba927b31260" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34831 } }
I20260926 16:37:33.005929 15361 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:33.006034 15361 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef783cf73fd40ebab052ba927b31260, State: Initialized, Role: FOLLOWER
I20260926 16:37:33.006574 15361 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef783cf73fd40ebab052ba927b31260" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34831 } }
I20260926 16:37:33.006789 15361 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:33.006910 15361 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:33.007048 15361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:33.009188 15361 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef783cf73fd40ebab052ba927b31260" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34831 } }
I20260926 16:37:33.009699 15361 leader_election.cc:302] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef783cf73fd40ebab052ba927b31260; no voters:
I20260926 16:37:33.010246 15361 leader_election.cc:288] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:33.010394 15366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:33.011492 15366 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [term 1 LEADER]: Becoming Leader. State: Replica: fef783cf73fd40ebab052ba927b31260, State: Running, Role: LEADER
I20260926 16:37:33.012049 15366 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef783cf73fd40ebab052ba927b31260" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34831 } }
I20260926 16:37:33.012440 15361 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:33.014779 15368 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef783cf73fd40ebab052ba927b31260. Latest consensus state: current_term: 1 leader_uuid: "fef783cf73fd40ebab052ba927b31260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef783cf73fd40ebab052ba927b31260" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34831 } } }
I20260926 16:37:33.014899 15367 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef783cf73fd40ebab052ba927b31260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef783cf73fd40ebab052ba927b31260" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34831 } } }
I20260926 16:37:33.015272 15367 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:33.015163 15368 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:33.016855 15375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:33.021612 15375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:33.029928 15375 catalog_manager.cc:1347] Generated new cluster ID: 2b40cf65a743404e988c0c6ff585a700
I20260926 16:37:33.030045 15375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:33.040377 15375 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:33.040868 15375 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:33.047253 15375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260: Generated new TSK 0
I20260926 16:37:33.048348 15375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:37:33.059089 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34831
--builtin_ntp_servers=127.10.110.20:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:37:33.375056 15385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:33.375344 15385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:33.375432 15385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:33.382839 15385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:33.382961 15385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:33.383071 15385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:33.393292 15385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:33.394572 15385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:33.396430 15385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:33.409289 15390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:33.409340 15391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:33.409487 15393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:33.410176 15385 server_base.cc:1034] running on GCE node
I20260926 16:37:33.410779 15385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:33.411691 15385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:33.412878 15385 hybrid_clock.cc:648] HybridClock initialized: now 1790440653412820 us; error 48 us; skew 500 ppm
I20260926 16:37:33.413160 15385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:33.415284 15385 webserver.cc:460] Webserver started at http://127.10.110.1:42397/ using document root <none> and password file <none>
I20260926 16:37:33.415973 15385 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:33.416059 15385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:33.416344 15385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:33.419090 15385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "9a6d4909f315499cab1ebefadce47416"
format_stamp: "Formatted at 2026-09-26 16:37:33 on dist-test-slave-whjp"
server_key: "23ed65122fe3e76811d774facf140b18"
server_key_iv: "7e96a96c12454ebb199859592288ed92"
server_key_version: "encryptionkey@0"
I20260926 16:37:33.420018 15385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "9a6d4909f315499cab1ebefadce47416"
format_stamp: "Formatted at 2026-09-26 16:37:33 on dist-test-slave-whjp"
server_key: "23ed65122fe3e76811d774facf140b18"
server_key_iv: "7e96a96c12454ebb199859592288ed92"
server_key_version: "encryptionkey@0"
I20260926 16:37:33.424929 15385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:37:33.428340 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:33.429612 15385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:33.429754 15385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "9a6d4909f315499cab1ebefadce47416"
format_stamp: "Formatted at 2026-09-26 16:37:33 on dist-test-slave-whjp"
server_key: "23ed65122fe3e76811d774facf140b18"
server_key_iv: "7e96a96c12454ebb199859592288ed92"
server_key_version: "encryptionkey@0"
I20260926 16:37:33.429975 15385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:33.435364 15385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:33.436170 15385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:33.436422 15385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:33.437259 15385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:33.438622 15385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:33.438712 15385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:33.438819 15385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:33.438887 15385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:33.468892 15385 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:36891
I20260926 16:37:33.468940 15511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:36891 every 8 connection(s)
I20260926 16:37:33.470189 15385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:33.476562 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15385
I20260926 16:37:33.476766 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:33.477466 10680 external_mini_cluster.cc:1428] Setting key 09c74f3805c9cd423bfd5ed0e53e2132
I20260926 16:37:33.477867 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:33.472708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:37445 (local address 127.10.110.62:34831)
0926 16:37:33.472867 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:33.472872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:33.474187 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 16:37:33.475761 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:33.475765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:33.475813 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:33.475929 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:33.477216 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:33.477221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:33.477226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:33.477247 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:33.477285 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:33.477300 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:33.477418 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:33.477550 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 16:37:33.477611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:37:33.478931 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:33.473159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34831 (local address 127.10.110.1:37445)
0926 16:37:33.474011 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:33.474051 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:33.475516 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:33.476136 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:33.476145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:33.476235 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:33.476889 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:33.476891 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:33.476929 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:33.477017 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:33.477384 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:33.477390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:33.477395 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:37:33.477608 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 16:37:33.477809 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
I20260926 16:37:33.480279 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34831
--builtin_ntp_servers=127.10.110.20:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 16:37:33.480389 15512 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34831
I20260926 16:37:33.480660 15512 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:33.481304 15512 heartbeater.cc:511] Master 127.10.110.62:34831 requested a full tablet report, sending...
I20260926 16:37:33.483358 15326 ts_manager.cc:194] Registered new tserver with Master: 9a6d4909f315499cab1ebefadce47416 (127.10.110.1:36891)
W20260926 16:37:33.794181 15515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:33.794486 15515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:33.794624 15515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:33.801759 15515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:33.801877 15515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:33.802030 15515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:33.811281 15515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:33.812621 15515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:33.814396 15515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:33.827013 15523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:33.826996 15521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:33.827005 15520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:33.827781 15515 server_base.cc:1034] running on GCE node
I20260926 16:37:33.828426 15515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:33.829352 15515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:33.830579 15515 hybrid_clock.cc:648] HybridClock initialized: now 1790440653830505 us; error 60 us; skew 500 ppm
I20260926 16:37:33.830915 15515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:33.833235 15515 webserver.cc:460] Webserver started at http://127.10.110.2:35081/ using document root <none> and password file <none>
I20260926 16:37:33.833981 15515 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:33.834090 15515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:33.834419 15515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:33.837406 15515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "cb66639631fd4d8093ec0400a62515ba"
format_stamp: "Formatted at 2026-09-26 16:37:33 on dist-test-slave-whjp"
server_key: "ae41955e6fc57c4a593f8dfa3d38f34a"
server_key_iv: "58198a7cb4595273fb7bae62a0d9f3e0"
server_key_version: "encryptionkey@0"
I20260926 16:37:33.838222 15515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "cb66639631fd4d8093ec0400a62515ba"
format_stamp: "Formatted at 2026-09-26 16:37:33 on dist-test-slave-whjp"
server_key: "ae41955e6fc57c4a593f8dfa3d38f34a"
server_key_iv: "58198a7cb4595273fb7bae62a0d9f3e0"
server_key_version: "encryptionkey@0"
I20260926 16:37:33.843705 15515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 16:37:33.847188 15529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:33.848574 15515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:33.848721 15515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "cb66639631fd4d8093ec0400a62515ba"
format_stamp: "Formatted at 2026-09-26 16:37:33 on dist-test-slave-whjp"
server_key: "ae41955e6fc57c4a593f8dfa3d38f34a"
server_key_iv: "58198a7cb4595273fb7bae62a0d9f3e0"
server_key_version: "encryptionkey@0"
I20260926 16:37:33.848927 15515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:33.854434 15515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:33.855196 15515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:33.855425 15515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:33.856356 15515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:33.857676 15515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:33.857759 15515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:33.857856 15515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:33.857928 15515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:33.886008 15515 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:40773
I20260926 16:37:33.886049 15641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:40773 every 8 connection(s)
I20260926 16:37:33.887334 15515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:33.894851 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:33.889738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:37251 (local address 127.10.110.62:34831)
0926 16:37:33.889940 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:33.889947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:33.891349 (+ 1402us) server_negotiation.cc:407] Connection header received
0926 16:37:33.893026 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:33.893030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:33.893086 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:33.893174 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:33.894312 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:33.894316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:33.894319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:33.894336 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:33.894366 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:33.894377 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:33.894465 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:33.894594 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 16:37:33.894638 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:37:33.895742 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:33.890301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34831 (local address 127.10.110.2:37251)
0926 16:37:33.891207 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:33.891243 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:33.892772 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:33.893360 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:33.893368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:33.893458 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:33.894037 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:33.894039 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:33.894075 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:33.894167 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:33.894447 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:33.894453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:33.894457 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:37:33.894669 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 16:37:33.894855 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":280,"threads_started":1}
I20260926 16:37:33.897258 15642 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34831
I20260926 16:37:33.897578 15642 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:33.897691 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15515
I20260926 16:37:33.897847 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:33.898277 15642 heartbeater.cc:511] Master 127.10.110.62:34831 requested a full tablet report, sending...
I20260926 16:37:33.898365 10680 external_mini_cluster.cc:1428] Setting key 846bbf7445ef56607315a7d01712d960
I20260926 16:37:33.899725 15325 ts_manager.cc:194] Registered new tserver with Master: cb66639631fd4d8093ec0400a62515ba (127.10.110.2:40773)
I20260926 16:37:33.901078 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34831
--builtin_ntp_servers=127.10.110.20:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:37:34.214169 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:34.214464 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:34.214551 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:34.221997 15645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:34.222106 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:34.222254 15645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:34.231932 15645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:34.233201 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:34.234998 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:34.248210 15650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:34.248237 15651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:34.248792 15653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:34.248889 15645 server_base.cc:1034] running on GCE node
I20260926 16:37:34.249488 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:34.250406 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:34.251598 15645 hybrid_clock.cc:648] HybridClock initialized: now 1790440654251541 us; error 50 us; skew 500 ppm
I20260926 16:37:34.251976 15645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:34.254246 15645 webserver.cc:460] Webserver started at http://127.10.110.3:35711/ using document root <none> and password file <none>
I20260926 16:37:34.254953 15645 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:34.255040 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:34.255350 15645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:34.258208 15645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "0f24b3e7c0204dffb679dc247e0b6f2d"
format_stamp: "Formatted at 2026-09-26 16:37:34 on dist-test-slave-whjp"
server_key: "ed26aa57559d7a810b81eaff100fdb61"
server_key_iv: "d7220d8e4aef9e217d7f4a24d376bb00"
server_key_version: "encryptionkey@0"
I20260926 16:37:34.258965 15645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "0f24b3e7c0204dffb679dc247e0b6f2d"
format_stamp: "Formatted at 2026-09-26 16:37:34 on dist-test-slave-whjp"
server_key: "ed26aa57559d7a810b81eaff100fdb61"
server_key_iv: "d7220d8e4aef9e217d7f4a24d376bb00"
server_key_version: "encryptionkey@0"
I20260926 16:37:34.264050 15645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:37:34.267421 15659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:34.268775 15645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:34.268921 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "0f24b3e7c0204dffb679dc247e0b6f2d"
format_stamp: "Formatted at 2026-09-26 16:37:34 on dist-test-slave-whjp"
server_key: "ed26aa57559d7a810b81eaff100fdb61"
server_key_iv: "d7220d8e4aef9e217d7f4a24d376bb00"
server_key_version: "encryptionkey@0"
I20260926 16:37:34.269119 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:34.274652 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:34.275408 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:34.275643 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:34.276538 15645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:34.277876 15645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:34.277966 15645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:34.278071 15645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:34.278137 15645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:34.306514 15645 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:33687
I20260926 16:37:34.306555 15771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:33687 every 8 connection(s)
I20260926 16:37:34.307790 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:34.308138 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15645
I20260926 16:37:34.308282 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:34.308835 10680 external_mini_cluster.cc:1428] Setting key c70c807d7fb750ab21abc0d53a25f14b
I20260926 16:37:34.315785 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.310495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:46523 (local address 127.10.110.62:34831)
0926 16:37:34.310693 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.310699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.312329 (+ 1630us) server_negotiation.cc:407] Connection header received
0926 16:37:34.313965 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.313970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.314018 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.314094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.315203 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.315207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.315210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.315226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.315258 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.315269 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.315360 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.315509 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.315551 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:37:34.316817 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.311074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34831 (local address 127.10.110.3:46523)
0926 16:37:34.312199 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.312244 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:34.313748 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:34.314280 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:34.314288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:34.314377 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:34.314948 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:34.314951 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:34.314987 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:34.315070 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:34.315342 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:34.315349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:34.315355 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:37:34.315572 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 16:37:34.315783 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":289,"threads_started":1}
I20260926 16:37:34.317996 15772 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34831
I20260926 16:37:34.318259 15772 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:34.318822 15772 heartbeater.cc:511] Master 127.10.110.62:34831 requested a full tablet report, sending...
I20260926 16:37:34.320060 15325 ts_manager.cc:194] Registered new tserver with Master: 0f24b3e7c0204dffb679dc247e0b6f2d (127.10.110.3:33687)
I20260926 16:37:34.324267 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:37:34.331221 15364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.328672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.10.110.62:34831)
0926 16:37:34.328830 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.328836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.329424 (+ 588us) server_negotiation.cc:407] Connection header received
0926 16:37:34.329629 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.329634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.329685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.329764 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.330654 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.330659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.330663 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.330679 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.330709 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.330721 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.330822 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.330904 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.330960 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:37:34.335292 15325 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:34.341907 15325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:34.360847 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.354142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.10.110.2:40773)
0926 16:37:34.355471 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.355481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.356061 (+ 580us) server_negotiation.cc:407] Connection header received
0926 16:37:34.356301 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.356311 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.356576 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.356713 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.359167 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.359179 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.359197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.359229 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.359295 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.359318 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.359484 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.360398 (+ 914us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.360499 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":1058,"threads_started":1}
I20260926 16:37:34.361382 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.353583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:40773 (local address 127.0.0.1:53038)
0926 16:37:34.355951 (+ 2368us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.355969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:34.356144 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:34.357878 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:34.357886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:34.357992 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:34.358849 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:34.358853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:34.358886 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:34.358981 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:34.359540 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:34.359550 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:34.359558 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 16:37:34.359746 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 16:37:34.359868 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2208,"thread_start_us":6211,"threads_started":1}
I20260926 16:37:34.362795 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.353675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.10.110.1:36891)
0926 16:37:34.359770 (+ 6095us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.359777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.359808 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:37:34.359894 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.359902 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.360108 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.360242 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.362094 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.362103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.362118 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.362137 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.362199 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.362226 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.362347 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.362412 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.362506 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5931,"thread_start_us":5801,"threads_started":1}
I20260926 16:37:34.362890 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.353522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53438)
0926 16:37:34.354167 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.354209 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:34.354330 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:34.361075 (+ 6745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:34.361080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:34.361105 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:34.361883 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:34.361887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:34.361903 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:34.361943 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:34.362308 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:34.362313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:34.362317 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:37:34.362432 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:37:34.362480 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260926 16:37:34.363510 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.355869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.10.110.3:33687)
0926 16:37:34.359330 (+ 3461us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.359339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.360215 (+ 876us) server_negotiation.cc:407] Connection header received
0926 16:37:34.360315 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.360322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.360510 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.360648 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.362337 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.362345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.362358 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.362376 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.362431 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.362449 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.362599 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.363070 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.363157 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3288,"thread_start_us":2865,"threads_started":1}
I20260926 16:37:34.364125 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.355284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:33687 (local address 127.0.0.1:44458)
0926 16:37:34.360073 (+ 4789us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.360099 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:34.360204 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:34.361450 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:34.361454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:34.361476 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:34.362096 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:34.362099 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:34.362112 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:34.362141 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:34.362649 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:34.362653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:34.362658 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:37:34.362769 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:37:34.362821 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4669,"thread_start_us":854,"threads_started":1}
I20260926 16:37:34.366365 15447 tablet_service.cc:1467] Processing CreateTablet for tablet 5d61b59938c146a38e00c5060c9eac39 (DEFAULT_TABLE table=test-table [id=eabfb26f432a4a9f81f461a261ea1d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:34.366199 15577 tablet_service.cc:1467] Processing CreateTablet for tablet 5d61b59938c146a38e00c5060c9eac39 (DEFAULT_TABLE table=test-table [id=eabfb26f432a4a9f81f461a261ea1d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:34.367640 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d61b59938c146a38e00c5060c9eac39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:34.367640 15447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d61b59938c146a38e00c5060c9eac39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:34.371119 15707 tablet_service.cc:1467] Processing CreateTablet for tablet 5d61b59938c146a38e00c5060c9eac39 (DEFAULT_TABLE table=test-table [id=eabfb26f432a4a9f81f461a261ea1d28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:34.372648 15707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d61b59938c146a38e00c5060c9eac39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:34.382164 15790 tablet_bootstrap.cc:492] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: Bootstrap starting.
I20260926 16:37:34.383553 15791 tablet_bootstrap.cc:492] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Bootstrap starting.
I20260926 16:37:34.386212 15792 tablet_bootstrap.cc:492] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Bootstrap starting.
I20260926 16:37:34.386633 15790 tablet_bootstrap.cc:654] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:34.387869 15791 tablet_bootstrap.cc:654] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:34.387970 15790 log.cc:824] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:34.388970 15791 log.cc:824] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:34.389626 15792 tablet_bootstrap.cc:654] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:34.390822 15792 log.cc:824] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:34.391109 15790 tablet_bootstrap.cc:492] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: No bootstrap required, opened a new log
I20260926 16:37:34.391400 15791 tablet_bootstrap.cc:492] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: No bootstrap required, opened a new log
I20260926 16:37:34.391388 15790 ts_tablet_manager.cc:1397] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 16:37:34.391738 15791 ts_tablet_manager.cc:1397] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260926 16:37:34.393689 15792 tablet_bootstrap.cc:492] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: No bootstrap required, opened a new log
I20260926 16:37:34.393993 15792 ts_tablet_manager.cc:1397] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 16:37:34.398983 15791 raft_consensus.cc:348] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.399340 15791 raft_consensus.cc:374] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:34.399451 15791 raft_consensus.cc:729] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb66639631fd4d8093ec0400a62515ba, State: Initialized, Role: FOLLOWER
I20260926 16:37:34.400058 15791 consensus_queue.cc:260] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.400996 15642 heartbeater.cc:503] Master 127.10.110.62:34831 was elected leader, sending a full tablet report...
I20260926 16:37:34.401506 15791 ts_tablet_manager.cc:1428] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 16:37:34.401233 15790 raft_consensus.cc:348] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.401629 15790 raft_consensus.cc:374] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:34.401737 15790 raft_consensus.cc:729] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6d4909f315499cab1ebefadce47416, State: Initialized, Role: FOLLOWER
I20260926 16:37:34.402298 15790 consensus_queue.cc:260] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.405634 15792 raft_consensus.cc:348] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.406095 15792 raft_consensus.cc:374] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:34.406204 15792 raft_consensus.cc:729] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f24b3e7c0204dffb679dc247e0b6f2d, State: Initialized, Role: FOLLOWER
I20260926 16:37:34.406822 15792 consensus_queue.cc:260] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.407881 15772 heartbeater.cc:503] Master 127.10.110.62:34831 was elected leader, sending a full tablet report...
I20260926 16:37:34.408633 15792 ts_tablet_manager.cc:1428] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260926 16:37:34.409040 15790 ts_tablet_manager.cc:1428] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
I20260926 16:37:34.409094 15512 heartbeater.cc:503] Master 127.10.110.62:34831 was elected leader, sending a full tablet report...
W20260926 16:37:34.472301 15513 tablet.cc:2369] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:34.497040 15798 raft_consensus.cc:482] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:34.497330 15798 raft_consensus.cc:504] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.498967 15798 leader_election.cc:288] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a6d4909f315499cab1ebefadce47416 (127.10.110.1:36891), cb66639631fd4d8093ec0400a62515ba (127.10.110.2:40773)
I20260926 16:37:34.501453 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.499372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.10.110.3:47839)
0926 16:37:34.499667 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.499682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:34.499781 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:34.500105 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:34.500110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:34.500135 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:34.500724 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:34.500726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:34.500740 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:34.500780 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:34.501047 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:34.501051 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:34.501056 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:37:34.501156 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 16:37:34.501206 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260926 16:37:34.501462 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.499417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:47839 (local address 127.10.110.1:36891)
0926 16:37:34.499593 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.499599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.499751 (+ 152us) server_negotiation.cc:407] Connection header received
0926 16:37:34.499920 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.499926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.499975 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.500055 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.500909 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.500914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.500919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.500933 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.500960 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.500971 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.501063 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.501159 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.501210 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:37:34.501914 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.499416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:36335 (local address 127.10.110.2:40773)
0926 16:37:34.499586 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.499592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.500274 (+ 682us) server_negotiation.cc:407] Connection header received
0926 16:37:34.500371 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.500377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.500425 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.500496 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.501293 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.501299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.501302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.501316 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.501341 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.501352 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.501471 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.501657 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.501713 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:37:34.502416 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.499452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:40773 (local address 127.10.110.3:36335)
0926 16:37:34.500170 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.500183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:34.500282 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:34.500539 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:34.500544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:34.500567 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:34.501102 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:37:34.501105 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:34.501117 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:37:34.501150 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:34.501524 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:34.501529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:34.501534 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:37:34.502087 (+ 553us) client_negotiation.cc:240] Negotiation successful
0926 16:37:34.502133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":404,"threads_started":1}
I20260926 16:37:34.502547 15467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d61b59938c146a38e00c5060c9eac39" candidate_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a6d4909f315499cab1ebefadce47416" is_pre_election: true
I20260926 16:37:34.503013 15467 raft_consensus.cc:2454] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f24b3e7c0204dffb679dc247e0b6f2d in term 0.
I20260926 16:37:34.503139 15597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d61b59938c146a38e00c5060c9eac39" candidate_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb66639631fd4d8093ec0400a62515ba" is_pre_election: true
I20260926 16:37:34.503525 15597 raft_consensus.cc:2454] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f24b3e7c0204dffb679dc247e0b6f2d in term 0.
I20260926 16:37:34.503619 15662 leader_election.cc:302] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f24b3e7c0204dffb679dc247e0b6f2d, 9a6d4909f315499cab1ebefadce47416; no voters:
I20260926 16:37:34.503963 15798 raft_consensus.cc:2781] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:34.504086 15798 raft_consensus.cc:482] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:34.504176 15798 raft_consensus.cc:3037] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:34.506207 15798 raft_consensus.cc:504] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.506750 15798 leader_election.cc:288] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [CANDIDATE]: Term 1 election: Requested vote from peers 9a6d4909f315499cab1ebefadce47416 (127.10.110.1:36891), cb66639631fd4d8093ec0400a62515ba (127.10.110.2:40773)
I20260926 16:37:34.507050 15467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d61b59938c146a38e00c5060c9eac39" candidate_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a6d4909f315499cab1ebefadce47416"
I20260926 16:37:34.507140 15597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d61b59938c146a38e00c5060c9eac39" candidate_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb66639631fd4d8093ec0400a62515ba"
I20260926 16:37:34.507208 15467 raft_consensus.cc:3037] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:34.507349 15597 raft_consensus.cc:3037] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:34.509307 15467 raft_consensus.cc:2454] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f24b3e7c0204dffb679dc247e0b6f2d in term 1.
I20260926 16:37:34.509479 15597 raft_consensus.cc:2454] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f24b3e7c0204dffb679dc247e0b6f2d in term 1.
I20260926 16:37:34.509624 15662 leader_election.cc:302] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f24b3e7c0204dffb679dc247e0b6f2d, 9a6d4909f315499cab1ebefadce47416; no voters:
I20260926 16:37:34.509878 15798 raft_consensus.cc:2781] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:34.510263 15798 raft_consensus.cc:686] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 1 LEADER]: Becoming Leader. State: Replica: 0f24b3e7c0204dffb679dc247e0b6f2d, State: Running, Role: LEADER
I20260926 16:37:34.510677 15798 consensus_queue.cc:237] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } }
I20260926 16:37:34.515121 15325 catalog_manager.cc:5495] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f24b3e7c0204dffb679dc247e0b6f2d (127.10.110.3). New cstate: current_term: 1 leader_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f24b3e7c0204dffb679dc247e0b6f2d" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33687 } health_report { overall_health: HEALTHY } } }
W20260926 16:37:34.559417 15773 tablet.cc:2369] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:34.562664 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.560891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.10.110.3:33687)
0926 16:37:34.561070 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.561074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.561106 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:37:34.561292 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.561295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.561372 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.561446 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.562229 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.562232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.562235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.562245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.562269 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.562279 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.562362 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.562396 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.562437 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:37:34.570184 15467 raft_consensus.cc:1261] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 1 FOLLOWER]: Refusing update from remote peer 0f24b3e7c0204dffb679dc247e0b6f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:34.570183 15597 raft_consensus.cc:1261] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 1 FOLLOWER]: Refusing update from remote peer 0f24b3e7c0204dffb679dc247e0b6f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:34.571089 15800 consensus_queue.cc:1035] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a6d4909f315499cab1ebefadce47416" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:34.571414 15798 consensus_queue.cc:1035] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb66639631fd4d8093ec0400a62515ba" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:34.594702 15325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:34.594923 15325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:34.598219 15325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fef783cf73fd40ebab052ba927b31260: Sending DeleteTablet for 3 replicas of tablet 5d61b59938c146a38e00c5060c9eac39
I20260926 16:37:34.599457 15577 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d61b59938c146a38e00c5060c9eac39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:34 UTC) from {username='slave'} at 127.0.0.1:53038
I20260926 16:37:34.599457 15447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d61b59938c146a38e00c5060c9eac39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:34 UTC) from {username='slave'} at 127.0.0.1:53438
I20260926 16:37:34.599898 15707 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d61b59938c146a38e00c5060c9eac39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:34 UTC) from {username='slave'} at 127.0.0.1:44458
I20260926 16:37:34.601284 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15385
I20260926 16:37:34.610744 15815 tablet_replica.cc:330] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: stopping tablet replica
I20260926 16:37:34.611214 15815 raft_consensus.cc:2229] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:34.611912 15815 raft_consensus.cc:2258] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:34.614220 15815 ts_tablet_manager.cc:1905] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:34.621654 15815 ts_tablet_manager.cc:1918] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:34.621878 15815 log.cc:1196] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/5d61b59938c146a38e00c5060c9eac39
I20260926 16:37:34.622435 15815 ts_tablet_manager.cc:1939] T 5d61b59938c146a38e00c5060c9eac39 P 0f24b3e7c0204dffb679dc247e0b6f2d: Deleting consensus metadata
I20260926 16:37:34.624749 15312 catalog_manager.cc:4841] TS 0f24b3e7c0204dffb679dc247e0b6f2d (127.10.110.3:33687): tablet 5d61b59938c146a38e00c5060c9eac39 (table test-table [id=eabfb26f432a4a9f81f461a261ea1d28]) successfully deleted
I20260926 16:37:34.628895 15814 tablet_replica.cc:330] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416: stopping tablet replica
I20260926 16:37:34.629508 15813 tablet_replica.cc:330] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: stopping tablet replica
I20260926 16:37:34.629695 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.602821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53452 (local address 127.10.110.1:36891)
0926 16:37:34.603001 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.603008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.627277 (+ 24269us) server_negotiation.cc:407] Connection header received
0926 16:37:34.627475 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.627481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.627594 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.627727 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.628900 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.628907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.628912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.628933 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.628972 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.628987 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.629086 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.629322 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.629386 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:37:34.630007 15814 raft_consensus.cc:2229] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:34.630059 15813 raft_consensus.cc:2229] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:34.630609 15814 raft_consensus.cc:2258] T 5d61b59938c146a38e00c5060c9eac39 P 9a6d4909f315499cab1ebefadce47416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:34.630774 15813 raft_consensus.cc:2258] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:34.630923 15507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:34.633411 15813 ts_tablet_manager.cc:1905] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:34.640847 15813 ts_tablet_manager.cc:1918] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:34.641026 15813 log.cc:1196] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/5d61b59938c146a38e00c5060c9eac39
I20260926 16:37:34.641569 15813 ts_tablet_manager.cc:1939] T 5d61b59938c146a38e00c5060c9eac39 P cb66639631fd4d8093ec0400a62515ba: Deleting consensus metadata
I20260926 16:37:34.643443 15311 catalog_manager.cc:4841] TS cb66639631fd4d8093ec0400a62515ba (127.10.110.2:40773): tablet 5d61b59938c146a38e00c5060c9eac39 (table test-table [id=eabfb26f432a4a9f81f461a261ea1d28]) successfully deleted
I20260926 16:37:34.719133 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15385
W20260926 16:37:34.732566 15312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.110.1:36891 (error 108)
I20260926 16:37:34.733207 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15515
I20260926 16:37:34.734480 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.733990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53464)
0926 16:37:34.734230 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.734347 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260926 16:37:34.734727 15312 catalog_manager.cc:4584] TS 9a6d4909f315499cab1ebefadce47416 (127.10.110.1:36891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d61b59938c146a38e00c5060c9eac39: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
I20260926 16:37:34.736496 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.734073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.10.110.2:40773)
0926 16:37:34.734251 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.734258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.734488 (+ 230us) server_negotiation.cc:407] Connection header received
0926 16:37:34.734702 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.734708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.734811 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.734883 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.735918 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.735922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.735925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.735941 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.735972 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.735984 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.736087 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.736188 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.736233 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:37:34.737169 15637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:34.788847 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.788196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53474)
0926 16:37:34.788440 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.788555 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 16:37:34.789994 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.789589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53484)
0926 16:37:34.789744 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.789852 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 16:37:34.823348 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15515
I20260926 16:37:34.836596 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15645
I20260926 16:37:34.840106 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.837766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.10.110.3:33687)
0926 16:37:34.838038 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:34.838058 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:34.838079 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:37:34.838188 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:34.838192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:34.838247 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:34.838457 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:34.839435 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:34.839440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:34.839445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:34.839462 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:34.839492 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:34.839504 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:34.839625 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:34.839717 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:37:34.839785 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:37:34.840868 15767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:34.850385 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.849954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53494)
0926 16:37:34.850142 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.850231 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 16:37:34.851279 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.850962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53500)
0926 16:37:34.851107 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.851171 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 16:37:34.931744 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.931307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53502)
0926 16:37:34.931506 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.931600 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 16:37:34.932766 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:34.932351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36891 (local address 127.0.0.1:53516)
0926 16:37:34.932527 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:34.932609 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:36891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 16:37:34.955390 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15645
I20260926 16:37:34.970170 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15295
I20260926 16:37:34.971319 15356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:35.075206 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15295
2026-09-26T16:37:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2565 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 16:37:35.111197 10680 mini_oidc.cc:152] Starting JWKS server
I20260926 16:37:35.113725 10680 webserver.cc:460] Webserver started at https://127.0.0.1:44021/ using document root <none> and password file <none>
I20260926 16:37:35.114022 10680 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:37:35.114620 10680 webserver.cc:460] Webserver started at http://127.0.0.1:39505/ using document root <none> and password file <none>
2026-09-26T16:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:35Z Disabled control of system clock
I20260926 16:37:35.138216 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33665
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33665
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44021/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/testchainca.cert with env {}
W20260926 16:37:35.451479 15827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:35.451767 15827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:35.451915 15827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:35.458935 15827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:35.459034 15827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:35.459079 15827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:35.459151 15827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:35.459205 15827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:37:35.459287 15827 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:37:35.459332 15827 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44021/jwks.json
I20260926 16:37:35.469089 15827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44021/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:35.470373 15827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:35.471673 15827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:35.482791 15835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:35.482836 15833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:35.483034 15832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:35.483700 15827 server_base.cc:1034] running on GCE node
I20260926 16:37:35.484457 15827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:35.485741 15827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:35.486927 15827 hybrid_clock.cc:648] HybridClock initialized: now 1790440655486891 us; error 25 us; skew 500 ppm
I20260926 16:37:35.487219 15827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:35.489279 15827 webserver.cc:460] Webserver started at http://127.10.110.62:32881/ using document root <none> and password file <none>
I20260926 16:37:35.489916 15827 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:35.490000 15827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:35.490252 15827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:35.493036 15827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "d708e79626eb47a08315d175193b40e8"
format_stamp: "Formatted at 2026-09-26 16:37:35 on dist-test-slave-whjp"
server_key: "cda2c5fe2eacd7ac52671163effbce69"
server_key_iv: "5968d7381bed646dc5e7b40660cec83f"
server_key_version: "encryptionkey@0"
I20260926 16:37:35.493733 15827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "d708e79626eb47a08315d175193b40e8"
format_stamp: "Formatted at 2026-09-26 16:37:35 on dist-test-slave-whjp"
server_key: "cda2c5fe2eacd7ac52671163effbce69"
server_key_iv: "5968d7381bed646dc5e7b40660cec83f"
server_key_version: "encryptionkey@0"
I20260926 16:37:35.498418 15827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 16:37:35.501881 15841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:35.503187 15827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:35.503328 15827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "d708e79626eb47a08315d175193b40e8"
format_stamp: "Formatted at 2026-09-26 16:37:35 on dist-test-slave-whjp"
server_key: "cda2c5fe2eacd7ac52671163effbce69"
server_key_iv: "5968d7381bed646dc5e7b40660cec83f"
server_key_version: "encryptionkey@0"
I20260926 16:37:35.503496 15827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:35.516556 15827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:35.517333 15827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:35.517611 15827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:35.534206 15827 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33665
I20260926 16:37:35.534233 15892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33665 every 8 connection(s)
I20260926 16:37:35.535547 15827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:35.539173 15893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:35.544909 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15827
I20260926 16:37:35.545090 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:35.545640 10680 external_mini_cluster.cc:1428] Setting key e788efd40486fd86784d3b49c5d1e443
I20260926 16:37:35.550127 15893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8: Bootstrap starting.
I20260926 16:37:35.553445 15893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:35.554493 15893 log.cc:824] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:35.557657 15893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8: No bootstrap required, opened a new log
I20260926 16:37:35.558650 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:35.547593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.10.110.62:33665)
0926 16:37:35.548608 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:35.548619 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:35.548657 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:37:35.550316 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:35.550340 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:35.550802 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:35.551138 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:35.551696 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:35.552997 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:35.554341 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:35.555022 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:35.555194 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:35.556399 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:35.556406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:35.556421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:35.556935 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:35.557008 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:35.557030 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:35.557118 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:35.557380 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 16:37:35.557692 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":353,"threads_started":1}
I20260926 16:37:35.562462 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:37:35.565903 15893 raft_consensus.cc:348] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d708e79626eb47a08315d175193b40e8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33665 } }
I20260926 16:37:35.566238 15893 raft_consensus.cc:374] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:35.566336 15893 raft_consensus.cc:729] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d708e79626eb47a08315d175193b40e8, State: Initialized, Role: FOLLOWER
I20260926 16:37:35.566880 15893 consensus_queue.cc:260] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d708e79626eb47a08315d175193b40e8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33665 } }
I20260926 16:37:35.567078 15893 raft_consensus.cc:388] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:35.567198 15893 raft_consensus.cc:482] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:35.567358 15893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:35.569483 15893 raft_consensus.cc:504] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d708e79626eb47a08315d175193b40e8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33665 } }
I20260926 16:37:35.569988 15893 leader_election.cc:302] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d708e79626eb47a08315d175193b40e8; no voters:
I20260926 16:37:35.570523 15893 leader_election.cc:288] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:35.570729 15898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:35.571695 15898 raft_consensus.cc:686] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [term 1 LEADER]: Becoming Leader. State: Replica: d708e79626eb47a08315d175193b40e8, State: Running, Role: LEADER
I20260926 16:37:35.572386 15898 consensus_queue.cc:237] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d708e79626eb47a08315d175193b40e8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33665 } }
I20260926 16:37:35.572574 15893 sys_catalog.cc:564] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:35.575167 15899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d708e79626eb47a08315d175193b40e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d708e79626eb47a08315d175193b40e8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33665 } } }
I20260926 16:37:35.575621 15899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:35.576470 15900 sys_catalog.cc:455] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d708e79626eb47a08315d175193b40e8. Latest consensus state: current_term: 1 leader_uuid: "d708e79626eb47a08315d175193b40e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d708e79626eb47a08315d175193b40e8" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33665 } } }
I20260926 16:37:35.576880 15900 sys_catalog.cc:458] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:35.578446 15908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:35.583191 15908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:35.591753 15908 catalog_manager.cc:1347] Generated new cluster ID: dd5ee4f790e14c8cbc49ce48f6b92e35
I20260926 16:37:35.591884 15908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:35.600871 15908 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:35.602114 15908 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:35.610909 15908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d708e79626eb47a08315d175193b40e8: Generated new TSK 0
I20260926 16:37:35.611693 15908 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:37:35.650540 15923 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 16:37:35.650689 15923 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:37:35.651880 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:35.630730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.10.110.62:33665)
0926 16:37:35.630918 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:35.630925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:35.631283 (+ 358us) server_negotiation.cc:407] Connection header received
0926 16:37:35.631537 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:35.631543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:35.631640 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:35.631798 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:37:35.632276 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:35.633405 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:35.634831 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:35.635482 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:35.635607 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:35.636125 (+ 518us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:37:35.651195 (+ 15070us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:37:35.651567 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51156: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 16:37:35.652000 15896 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51156: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 16:37:35.653177 15922 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:35.630592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33665 (local address 127.0.0.1:51156)
0926 16:37:35.631187 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:35.631203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:35.631343 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:35.631805 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:35.631809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:35.631818 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:37:35.632143 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:35.632153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:35.633524 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:35.633527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:35.634680 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:35.634694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:35.635634 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:35.635638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:35.635765 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:35.635824 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:37:35.651469 (+ 15645us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:37:35.651745 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.110.62:33665: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":282,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.110.62:33665: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 16:37:35.655086 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15827
I20260926 16:37:35.656231 15888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:35.747983 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15827
2026-09-26T16:37:35Z chronyd exiting
I20260926 16:37:35.794490 10680 test_util.cc:175] -----------------------------------------------
I20260926 16:37:35.794626 10680 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440615325743-10680-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (687 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 16:37:35.798401 10680 mini_oidc.cc:152] Starting JWKS server
I20260926 16:37:35.800194 10680 webserver.cc:460] Webserver started at https://127.0.0.1:45519/ using document root <none> and password file <none>
I20260926 16:37:35.800408 10680 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:37:35.801035 10680 webserver.cc:460] Webserver started at http://127.0.0.1:36277/ using document root <none> and password file <none>
2026-09-26T16:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:35Z Disabled control of system clock
I20260926 16:37:35.824702 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:46063
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:46063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45519/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/testchainca.cert with env {}
W20260926 16:37:36.135329 15930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:36.135627 15930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:36.135740 15930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:36.142912 15930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:36.143023 15930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:36.143067 15930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:36.143121 15930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:36.143169 15930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:37:36.143227 15930 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:37:36.143263 15930 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45519/jwks.json
I20260926 16:37:36.153057 15930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:46063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:46063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45519/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:36.154336 15930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:36.155570 15930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:36.166702 15935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:36.166672 15938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:36.166738 15936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:36.167810 15930 server_base.cc:1034] running on GCE node
I20260926 16:37:36.168668 15930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:36.170147 15930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:36.171365 15930 hybrid_clock.cc:648] HybridClock initialized: now 1790440656171319 us; error 34 us; skew 500 ppm
I20260926 16:37:36.171769 15930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:36.174222 15930 webserver.cc:460] Webserver started at http://127.10.110.62:34697/ using document root <none> and password file <none>
I20260926 16:37:36.174929 15930 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:36.175035 15930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:36.175365 15930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:36.178400 15930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0"
format_stamp: "Formatted at 2026-09-26 16:37:36 on dist-test-slave-whjp"
server_key: "46d392bc2dbbcee0bfa029d553d4f4a8"
server_key_iv: "cdcb441627dcc29ca262206e835041e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:36.179351 15930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0"
format_stamp: "Formatted at 2026-09-26 16:37:36 on dist-test-slave-whjp"
server_key: "46d392bc2dbbcee0bfa029d553d4f4a8"
server_key_iv: "cdcb441627dcc29ca262206e835041e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:36.184477 15930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:37:36.187736 15944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:36.189035 15930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:36.189229 15930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0"
format_stamp: "Formatted at 2026-09-26 16:37:36 on dist-test-slave-whjp"
server_key: "46d392bc2dbbcee0bfa029d553d4f4a8"
server_key_iv: "cdcb441627dcc29ca262206e835041e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:36.189422 15930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:36.222649 15930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:36.223472 15930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:36.223759 15930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:36.240661 15930 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:46063
I20260926 16:37:36.240732 15995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:46063 every 8 connection(s)
I20260926 16:37:36.242023 15930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:36.245694 15996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:36.251515 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 15930
I20260926 16:37:36.251690 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:36.252324 10680 external_mini_cluster.cc:1428] Setting key 6cf9b8960791e4ca958a03ff79fede82
I20260926 16:37:36.256227 15996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0: Bootstrap starting.
I20260926 16:37:36.259413 15996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:36.260666 15996 log.cc:824] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:36.263972 15996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0: No bootstrap required, opened a new log
I20260926 16:37:36.265897 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:36.254241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.10.110.62:46063)
0926 16:37:36.255176 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:36.255188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:36.255230 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:37:36.256906 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:36.256930 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:36.257384 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:36.257730 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:36.258307 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.259541 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.261346 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.262306 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.262437 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:36.263574 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:36.263581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:36.263596 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:36.264189 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:36.264267 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:36.264289 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:36.264382 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:36.264631 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 16:37:36.264956 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":317,"threads_started":1}
I20260926 16:37:36.269425 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:37:36.271564 15996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 46063 } }
I20260926 16:37:36.271941 15996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:36.272032 15996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa2eb46a7184ddea04cf77d45a4c4d0, State: Initialized, Role: FOLLOWER
I20260926 16:37:36.272593 15996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 46063 } }
I20260926 16:37:36.272805 15996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:36.272925 15996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:36.273063 15996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:36.275118 15996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 46063 } }
I20260926 16:37:36.275629 15996 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa2eb46a7184ddea04cf77d45a4c4d0; no voters:
I20260926 16:37:36.276213 15996 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:36.276355 16001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:36.276996 16001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa2eb46a7184ddea04cf77d45a4c4d0, State: Running, Role: LEADER
I20260926 16:37:36.277529 16001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 46063 } }
I20260926 16:37:36.278092 15996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:36.280084 16002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 46063 } } }
I20260926 16:37:36.280537 16002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:36.280407 16003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa2eb46a7184ddea04cf77d45a4c4d0. Latest consensus state: current_term: 1 leader_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa2eb46a7184ddea04cf77d45a4c4d0" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 46063 } } }
I20260926 16:37:36.280707 16003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:36.281328 16009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:36.286700 16009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:36.295627 16009 catalog_manager.cc:1347] Generated new cluster ID: d2f56a6bbb6b46b5bbee21eff65118cf
I20260926 16:37:36.295732 16009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:36.309171 16009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:36.310658 16009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:36.322552 16009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa2eb46a7184ddea04cf77d45a4c4d0: Generated new TSK 0
I20260926 16:37:36.323428 16009 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:37:36.365005 16026 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 16:37:36.365110 16026 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:37:36.366299 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:36.347096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.10.110.62:46063)
0926 16:37:36.347274 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:36.347281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:36.347638 (+ 357us) server_negotiation.cc:407] Connection header received
0926 16:37:36.347879 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:36.347884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:36.347985 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:36.348142 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:37:36.348584 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.349707 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.350944 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.351593 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.351720 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:36.352158 (+ 438us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:37:36.365670 (+ 13512us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:37:36.366028 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 16:37:36.366364 16025 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:36.346968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:46063 (local address 127.0.0.1:51584)
0926 16:37:36.347551 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:36.347567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:36.347680 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:36.348142 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:36.348145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:36.348154 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:37:36.348432 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:36.348442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.349818 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.349822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:36.350823 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:36.350833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.351731 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.351735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:36.351888 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:36.351921 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:37:36.365879 (+ 13958us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:37:36.366075 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.110.62:46063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":335,"threads_started":1}
W20260926 16:37:36.366412 15999 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45519/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 16:37:36.367486 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 15930
I20260926 16:37:36.368698 15991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:36.466459 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 15930
2026-09-26T16:37:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 16:37:36.518054 10680 mini_oidc.cc:152] Starting JWKS server
I20260926 16:37:36.519867 10680 webserver.cc:460] Webserver started at https://127.0.0.1:37623/ using document root <none> and password file <none>
I20260926 16:37:36.520107 10680 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:37:36.520712 10680 webserver.cc:460] Webserver started at http://127.0.0.1:37705/ using document root <none> and password file <none>
2026-09-26T16:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:36Z Disabled control of system clock
I20260926 16:37:36.544070 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:42991
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:42991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:37:36.842330 16033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:36.842603 16033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:36.842721 16033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:36.849644 16033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:36.849757 16033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:36.849818 16033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:36.849912 16033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:36.849970 16033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:37:36.850056 16033 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:37:36.850100 16033 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37623/jwks.json
I20260926 16:37:36.859537 16033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:42991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:36.860869 16033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:36.862646 16033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:36.873704 16038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:36.873714 16039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:36.873746 16041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:36.874585 16033 server_base.cc:1034] running on GCE node
I20260926 16:37:36.875293 16033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:36.876626 16033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:36.877817 16033 hybrid_clock.cc:648] HybridClock initialized: now 1790440656877759 us; error 43 us; skew 500 ppm
I20260926 16:37:36.878142 16033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:36.880255 16033 webserver.cc:460] Webserver started at http://127.10.110.62:45963/ using document root <none> and password file <none>
I20260926 16:37:36.880903 16033 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:36.880995 16033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:36.881286 16033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:36.884027 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "40cc43ffa9254fd59cfcaef6d42558fb"
format_stamp: "Formatted at 2026-09-26 16:37:36 on dist-test-slave-whjp"
server_key: "9a5e1abeb7721cb79a2ca8778808a426"
server_key_iv: "eff3570d178796e689f56a3d1908abbb"
server_key_version: "encryptionkey@0"
I20260926 16:37:36.884867 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "40cc43ffa9254fd59cfcaef6d42558fb"
format_stamp: "Formatted at 2026-09-26 16:37:36 on dist-test-slave-whjp"
server_key: "9a5e1abeb7721cb79a2ca8778808a426"
server_key_iv: "eff3570d178796e689f56a3d1908abbb"
server_key_version: "encryptionkey@0"
I20260926 16:37:36.889870 16033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:37:36.893059 16047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:36.894305 16033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260926 16:37:36.894455 16033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "40cc43ffa9254fd59cfcaef6d42558fb"
format_stamp: "Formatted at 2026-09-26 16:37:36 on dist-test-slave-whjp"
server_key: "9a5e1abeb7721cb79a2ca8778808a426"
server_key_iv: "eff3570d178796e689f56a3d1908abbb"
server_key_version: "encryptionkey@0"
I20260926 16:37:36.894637 16033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:36.933845 16033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:36.934726 16033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:36.935011 16033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:36.951792 16033 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:42991
I20260926 16:37:36.951812 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:42991 every 8 connection(s)
I20260926 16:37:36.953116 16033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:36.956604 16099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:36.959055 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16033
I20260926 16:37:36.959307 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:36.959973 10680 external_mini_cluster.cc:1428] Setting key b07430949d58369db006825da2228e0c
I20260926 16:37:36.968745 16099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb: Bootstrap starting.
I20260926 16:37:36.972541 16099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:36.973232 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:36.961662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.10.110.62:42991)
0926 16:37:36.962633 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:36.962644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:36.962683 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:37:36.964282 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:36.964304 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:36.964751 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:36.965076 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:36.965709 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.967001 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.968925 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:36.969590 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:36.969720 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:36.970927 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:36.970938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:36.970951 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:36.971455 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:36.971535 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:36.971555 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:36.971641 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:36.971896 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 16:37:36.972200 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":338,"threads_started":1}
I20260926 16:37:36.973737 16099 log.cc:824] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:36.976964 16099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb: No bootstrap required, opened a new log
I20260926 16:37:36.977255 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:37:36.985421 16099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42991 } }
I20260926 16:37:36.985826 16099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:36.985931 16099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cc43ffa9254fd59cfcaef6d42558fb, State: Initialized, Role: FOLLOWER
I20260926 16:37:36.986507 16099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42991 } }
I20260926 16:37:36.986733 16099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:36.986864 16099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:36.987031 16099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:36.989845 16099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42991 } }
I20260926 16:37:36.990413 16099 leader_election.cc:302] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40cc43ffa9254fd59cfcaef6d42558fb; no voters:
I20260926 16:37:36.990967 16099 leader_election.cc:288] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:36.991065 16104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:36.992002 16104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [term 1 LEADER]: Becoming Leader. State: Replica: 40cc43ffa9254fd59cfcaef6d42558fb, State: Running, Role: LEADER
I20260926 16:37:36.992496 16104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42991 } }
I20260926 16:37:36.992960 16099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:36.995174 16105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42991 } } }
I20260926 16:37:36.995549 16105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:36.995918 16106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40cc43ffa9254fd59cfcaef6d42558fb. Latest consensus state: current_term: 1 leader_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cc43ffa9254fd59cfcaef6d42558fb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 42991 } } }
I20260926 16:37:36.996897 16106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:36.997723 16111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:37.002753 16111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:37.011262 16111 catalog_manager.cc:1347] Generated new cluster ID: 871fb67a61ad410fbe69ad1643eda635
I20260926 16:37:37.011371 16111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:37.057598 16111 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:37.059401 16111 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:37.075775 16111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40cc43ffa9254fd59cfcaef6d42558fb: Generated new TSK 0
I20260926 16:37:37.076589 16111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:37:37.110810 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:37.101956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.10.110.62:42991)
0926 16:37:37.102302 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:37.102310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:37.102948 (+ 638us) server_negotiation.cc:407] Connection header received
0926 16:37:37.103144 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:37.103150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:37.103247 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:37.103422 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:37:37.103861 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.105007 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.106324 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.107014 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.107165 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:37.107859 (+ 694us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:37:37.110145 (+ 2286us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:37:37.110485 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":163}
W20260926 16:37:37.111115 16128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:37.101945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:42991 (local address 127.0.0.1:43222)
0926 16:37:37.102843 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:37.102858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:37.102984 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:37.103415 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:37.103418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:37.103426 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:37:37.103682 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:37.103690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.105174 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.105178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:37.106193 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:37.106204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.107309 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.107317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:37.107514 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:37.107559 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:37:37.110368 (+ 2809us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:37:37.110655 (+ 287us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.110.62:42991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":616,"threads_started":1}
W20260926 16:37:37.111187 16102 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 16:37:37.112398 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16033
I20260926 16:37:37.113554 16094 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 16:37:37.124462 16129 webserver.cc:533] Webserver: sslize failed:
E20260926 16:37:37.124557 16129 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:37:37.208092 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16033
2026-09-26T16:37:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T16:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:37Z Disabled control of system clock
I20260926 16:37:37.280885 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38705
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:36901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:38705
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/testchainca.cert with env {}
W20260926 16:37:37.581451 16134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:37.581735 16134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:37.581857 16134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:37.589025 16134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:37.589134 16134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:37.589191 16134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:37.589272 16134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:37.589332 16134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:37:37.589411 16134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:37:37.589457 16134 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 16:37:37.599305 16134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:38705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:37.600557 16134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:37.601796 16134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:37.612727 16139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:37.612690 16140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:37.612800 16142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:37.613312 16134 server_base.cc:1034] running on GCE node
I20260926 16:37:37.614020 16134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:37.615319 16134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:37.616537 16134 hybrid_clock.cc:648] HybridClock initialized: now 1790440657616484 us; error 42 us; skew 500 ppm
I20260926 16:37:37.616907 16134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:37:37.619120 16134 webserver.cc:460] Webserver started at http://127.10.110.62:32923/ using document root <none> and password file <none>
I20260926 16:37:37.619807 16134 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:37.619944 16134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:37.620231 16134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:37.622987 16134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6"
format_stamp: "Formatted at 2026-09-26 16:37:37 on dist-test-slave-whjp"
server_key: "3b7e254a5ed61dad2fed03010bd56033"
server_key_iv: "21c4091222de7ca38d56554a7de39f0a"
server_key_version: "encryptionkey@0"
I20260926 16:37:37.623721 16134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6"
format_stamp: "Formatted at 2026-09-26 16:37:37 on dist-test-slave-whjp"
server_key: "3b7e254a5ed61dad2fed03010bd56033"
server_key_iv: "21c4091222de7ca38d56554a7de39f0a"
server_key_version: "encryptionkey@0"
I20260926 16:37:37.628619 16134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:37:37.631907 16148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:37.633301 16134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:37:37.633435 16134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6"
format_stamp: "Formatted at 2026-09-26 16:37:37 on dist-test-slave-whjp"
server_key: "3b7e254a5ed61dad2fed03010bd56033"
server_key_iv: "21c4091222de7ca38d56554a7de39f0a"
server_key_version: "encryptionkey@0"
I20260926 16:37:37.633607 16134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:37.651690 16134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:37.652525 16134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:37.652755 16134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:37.668853 16134 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:38705
I20260926 16:37:37.668874 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:38705 every 8 connection(s)
I20260926 16:37:37.670167 16134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:37.673871 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:37.675655 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16134
I20260926 16:37:37.675809 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:37.676460 10680 external_mini_cluster.cc:1428] Setting key 11540f6074fc378705c7292b21ff4a19
I20260926 16:37:37.684567 16200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6: Bootstrap starting.
I20260926 16:37:37.687477 16200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:37.688531 16200 log.cc:824] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:37.688828 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:37.678283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.10.110.62:38705)
0926 16:37:37.679212 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:37.679223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:37.679263 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:37:37.680865 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:37.680888 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:37.681338 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:37.681688 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:37.682250 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.683507 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.684743 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.685367 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.685521 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:37.686631 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:37.686638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:37.686651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:37:37.687155 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:37.687243 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:37:37.687262 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:37.687352 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:37.687562 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 16:37:37.687895 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":279,"threads_started":1}
I20260926 16:37:37.691233 16200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6: No bootstrap required, opened a new log
I20260926 16:37:37.692351 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:37:37.698556 16200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38705 } }
I20260926 16:37:37.698892 16200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:37.698976 16200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1158e5d34b49dabe5a27c3a1bbd5b6, State: Initialized, Role: FOLLOWER
I20260926 16:37:37.699472 16200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38705 } }
I20260926 16:37:37.699669 16200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:37.699791 16200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:37.699961 16200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:37.701958 16200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38705 } }
I20260926 16:37:37.702427 16200 leader_election.cc:302] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f1158e5d34b49dabe5a27c3a1bbd5b6; no voters:
I20260926 16:37:37.702936 16200 leader_election.cc:288] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:37.703058 16205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:37.704105 16205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1158e5d34b49dabe5a27c3a1bbd5b6, State: Running, Role: LEADER
I20260926 16:37:37.704568 16205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38705 } }
I20260926 16:37:37.704994 16200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:37.707052 16207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f1158e5d34b49dabe5a27c3a1bbd5b6. Latest consensus state: current_term: 1 leader_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38705 } } }
I20260926 16:37:37.707362 16207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:37.707306 16206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1158e5d34b49dabe5a27c3a1bbd5b6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38705 } } }
I20260926 16:37:37.707630 16206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:37.709128 16214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:37.716990 16214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:37.725875 16214 catalog_manager.cc:1347] Generated new cluster ID: 3cd4c33430b049a285b5208c2e18f2d3
I20260926 16:37:37.726012 16214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:37.740269 16214 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:37.741520 16214 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:37.746958 16214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f1158e5d34b49dabe5a27c3a1bbd5b6: Generated new TSK 0
I20260926 16:37:37.747741 16214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:37:38.272183 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:37.758750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.10.110.62:38705)
0926 16:37:37.758952 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:37.758959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:37.759572 (+ 613us) server_negotiation.cc:407] Connection header received
0926 16:37:37.759780 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:37.759786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:37.759907 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:37.760062 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:37:37.760522 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.761615 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.762971 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.763680 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.763803 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:37.764303 (+ 500us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:37:38.271604 (+507301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:37:38.271962 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42046: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 16:37:38.272301 16203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42046: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 16:37:38.272684 16229 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:37.758763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:38705 (local address 127.0.0.1:42046)
0926 16:37:37.759471 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:37.759489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:37.759616 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:37.760118 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:37.760122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:37.760133 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:37:37.760368 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:37.760376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.761777 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.761781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:37.762841 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:37.762852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:37.763891 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:37.763898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:37.764030 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:37.764063 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:37:38.271907 (+507844us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:37:38.272138 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.110.62:38705: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":332,"threads_started":1}
I20260926 16:37:38.273758 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16134
I20260926 16:37:38.274813 16195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:38.374706 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16134
2026-09-26T16:37:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1148 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (395 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (411 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:39 dist-test-slave-whjp krb5kdc[16244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:39 dist-test-slave-whjp krb5kdc[16244](info): set up 2 sockets
Sep 26 16:37:39 dist-test-slave-whjp krb5kdc[16244](info): commencing operation
krb5kdc: starting...
W20260926 16:37:40.177258 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:40 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:40.385244 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33933
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:40.690529 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:40.690851 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:40.690982 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:40.698478 16257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:40.698599 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:40.698678 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:40.698769 16257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:40.698825 16257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:40.708600 16257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:40.709937 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:40.711215 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:40.722203 16262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:40.722277 16263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:40.722508 16265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:40.723321 16257 server_base.cc:1034] running on GCE node
I20260926 16:37:40.724098 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:40.725466 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:40.726670 16257 hybrid_clock.cc:648] HybridClock initialized: now 1790440660726615 us; error 44 us; skew 500 ppm
I20260926 16:37:40.726989 16257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:40 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440660, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:40.734148 16257 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:37:40.735546 16257 webserver.cc:460] Webserver started at http://127.10.110.62:43903/ using document root <none> and password file <none>
I20260926 16:37:40.736227 16257 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:40.736325 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:40.736579 16257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:40.739368 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "808caaf1f82b4d9094596983ec137f6b"
format_stamp: "Formatted at 2026-09-26 16:37:40 on dist-test-slave-whjp"
server_key: "802a77ab19e706b37c0eb625d0e7eddc"
server_key_iv: "31de16326ca464806072a132f8b82469"
server_key_version: "encryptionkey@0"
I20260926 16:37:40.740103 16257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "808caaf1f82b4d9094596983ec137f6b"
format_stamp: "Formatted at 2026-09-26 16:37:40 on dist-test-slave-whjp"
server_key: "802a77ab19e706b37c0eb625d0e7eddc"
server_key_iv: "31de16326ca464806072a132f8b82469"
server_key_version: "encryptionkey@0"
I20260926 16:37:40.744838 16257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:37:40.748003 16272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:40.749231 16257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:37:40.749362 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "808caaf1f82b4d9094596983ec137f6b"
format_stamp: "Formatted at 2026-09-26 16:37:40 on dist-test-slave-whjp"
server_key: "802a77ab19e706b37c0eb625d0e7eddc"
server_key_iv: "31de16326ca464806072a132f8b82469"
server_key_version: "encryptionkey@0"
I20260926 16:37:40.749521 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:40.776129 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:40.780310 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:40.780541 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:40.797061 16257 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33933
I20260926 16:37:40.797082 16323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33933 every 8 connection(s)
I20260926 16:37:40.798498 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:40.802135 16324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:40.803035 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16257
I20260926 16:37:40.803233 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:40.803922 10680 external_mini_cluster.cc:1428] Setting key aa005d8133cd2c9956249c0ffacdc7f6
I20260926 16:37:40.816766 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b: Bootstrap starting.
Sep 26 16:37:40 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:40.820904 16324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:40.822270 16324 log.cc:824] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:40.826699 16324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b: No bootstrap required, opened a new log
I20260926 16:37:40.831574 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:40.805920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.10.110.62:33933)
0926 16:37:40.806769 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:40.806784 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:40.806831 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:37:40.808561 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:40.808591 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:40.809031 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:40.809374 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:40.811374 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:40.813516 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:40.815462 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:40.816550 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:40.816736 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:40.821967 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:40.821978 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:40.821998 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:40.822051 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:40.826056 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:40.826914 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:40.826922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:40.826931 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:40.827141 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:40.827643 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:40.827648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:40.827652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:40.828364 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:40.828657 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:40.829664 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 16:37:40.830158 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":385,"threads_started":1}
I20260926 16:37:40.837695 16324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808caaf1f82b4d9094596983ec137f6b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33933 } }
I20260926 16:37:40.838169 16324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:40.838280 16324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808caaf1f82b4d9094596983ec137f6b, State: Initialized, Role: FOLLOWER
I20260926 16:37:40.838953 16324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808caaf1f82b4d9094596983ec137f6b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33933 } }
I20260926 16:37:40.839257 16324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:40.839390 16324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:40.839562 16324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:40.842978 16324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808caaf1f82b4d9094596983ec137f6b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33933 } }
I20260926 16:37:40.843562 16324 leader_election.cc:302] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 808caaf1f82b4d9094596983ec137f6b; no voters:
I20260926 16:37:40.844283 16324 leader_election.cc:288] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:40.844455 16329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:40.845101 16329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [term 1 LEADER]: Becoming Leader. State: Replica: 808caaf1f82b4d9094596983ec137f6b, State: Running, Role: LEADER
I20260926 16:37:40.846282 16329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808caaf1f82b4d9094596983ec137f6b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33933 } }
I20260926 16:37:40.846556 16324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:40.849308 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808caaf1f82b4d9094596983ec137f6b. Latest consensus state: current_term: 1 leader_uuid: "808caaf1f82b4d9094596983ec137f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808caaf1f82b4d9094596983ec137f6b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33933 } } }
I20260926 16:37:40.849282 16330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "808caaf1f82b4d9094596983ec137f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808caaf1f82b4d9094596983ec137f6b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33933 } } }
I20260926 16:37:40.849705 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:40.849705 16330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:40.856946 16337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:40.862761 16337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:40.873889 16337 catalog_manager.cc:1347] Generated new cluster ID: 23263e158cc64844815f34da0d0f6049
I20260926 16:37:40.874053 16337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:40.892798 16337 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:40.894117 16337 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:40.899277 16337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 808caaf1f82b4d9094596983ec137f6b: Generated new TSK 0
I20260926 16:37:40.900369 16337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:37:40.951416 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33933
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:41.268107 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:41.268385 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:41.268518 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:41.275553 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:41.275660 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:41.275810 16350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:41.285616 16350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:41.286871 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:41.288195 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:41.300336 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:41.300460 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:41.300504 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:41.301399 16350 server_base.cc:1034] running on GCE node
I20260926 16:37:41.301985 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:41.302878 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:41.304076 16350 hybrid_clock.cc:648] HybridClock initialized: now 1790440661304026 us; error 41 us; skew 500 ppm
I20260926 16:37:41.304388 16350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:41 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440661, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:41.307292 16350 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:37:41.308736 16350 webserver.cc:460] Webserver started at http://127.10.110.1:38003/ using document root <none> and password file <none>
I20260926 16:37:41.309401 16350 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:41.309517 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:41.309795 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:41.312644 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "0499ae5fc9594e4b8e3ff52134a6b139"
format_stamp: "Formatted at 2026-09-26 16:37:41 on dist-test-slave-whjp"
server_key: "bcf7d175293145d82211bcd8e50db472"
server_key_iv: "673edce7e87d8caa60e723d2bdd84a28"
server_key_version: "encryptionkey@0"
I20260926 16:37:41.313364 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "0499ae5fc9594e4b8e3ff52134a6b139"
format_stamp: "Formatted at 2026-09-26 16:37:41 on dist-test-slave-whjp"
server_key: "bcf7d175293145d82211bcd8e50db472"
server_key_iv: "673edce7e87d8caa60e723d2bdd84a28"
server_key_version: "encryptionkey@0"
I20260926 16:37:41.318274 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:37:41.321635 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:41.322903 16350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:37:41.323048 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "0499ae5fc9594e4b8e3ff52134a6b139"
format_stamp: "Formatted at 2026-09-26 16:37:41 on dist-test-slave-whjp"
server_key: "bcf7d175293145d82211bcd8e50db472"
server_key_iv: "673edce7e87d8caa60e723d2bdd84a28"
server_key_version: "encryptionkey@0"
I20260926 16:37:41.323233 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:41.342243 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:41.346078 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:41.346371 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:41.347265 16350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:41.348698 16350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:41.348773 16350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:41.348884 16350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:41.348932 16350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:41.376924 16350 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:46371
I20260926 16:37:41.376994 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:46371 every 8 connection(s)
I20260926 16:37:41.378090 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:41.381037 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16350
I20260926 16:37:41.381187 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:41.381934 10680 external_mini_cluster.cc:1428] Setting key 96ddfb5f031b6ff2083b96f2cf279e58
Sep 26 16:37:41 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440661, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:41.398540 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:41.380493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:56805 (local address 127.10.110.62:33933)
0926 16:37:41.380956 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:41.380963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:41.383893 (+ 2930us) server_negotiation.cc:407] Connection header received
0926 16:37:41.384206 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:41.384210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:41.384269 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:41.384355 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:41.386198 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.386891 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.388897 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.389289 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.389384 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:41.393264 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:41.393269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:41.393272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:41.393303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:41.395586 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:41.396384 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:41.396388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:41.396390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:41.396496 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:41.396915 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:41.396918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:41.396919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:41.397161 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:41.397259 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:41.398067 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 16:37:41.398225 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260926 16:37:41.399981 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:41.381065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33933 (local address 127.10.110.1:56805)
0926 16:37:41.381914 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:41.381970 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:41.383911 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:41.384617 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:41.384628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:41.385285 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:41.386036 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:41.386057 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.387019 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.387024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:41.388743 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:41.388768 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.389495 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.389502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:41.389761 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:41.390482 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:41.390510 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:41.393049 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:41.395704 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:41.395712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:41.395738 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:41.396282 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:41.396582 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:41.396587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:41.396590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:41.396823 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:41.397267 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:41.397275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:41.397684 (+ 409us) client_negotiation.cc:769] Sending connection context
0926 16:37:41.397996 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 16:37:41.398535 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":316,"threads_started":1}
I20260926 16:37:41.401363 16478 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33933
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:37:41.401676 16478 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:41.402405 16478 heartbeater.cc:511] Master 127.10.110.62:33933 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:37:41.404388 16289 ts_manager.cc:194] Registered new tserver with Master: 0499ae5fc9594e4b8e3ff52134a6b139 (127.10.110.1:46371)
I20260926 16:37:41.405732 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:56805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:37:41.426071 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33933
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:41.729401 16484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:41.729655 16484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:41.729744 16484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:41.736712 16484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:41.736807 16484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:41.736922 16484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:41.746337 16484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:41.747681 16484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:41.748992 16484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:41.761348 16489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:41.761464 16492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:41.761372 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:41.762020 16484 server_base.cc:1034] running on GCE node
I20260926 16:37:41.762588 16484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:41.763448 16484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:41.764652 16484 hybrid_clock.cc:648] HybridClock initialized: now 1790440661764582 us; error 53 us; skew 500 ppm
I20260926 16:37:41.764950 16484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:41 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440661, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:41.767871 16484 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:37:41.769260 16484 webserver.cc:460] Webserver started at http://127.10.110.2:33911/ using document root <none> and password file <none>
I20260926 16:37:41.769905 16484 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:41.770009 16484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:41.770301 16484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:41.773136 16484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "4edbc8b888144b49bc15da38a29681bd"
format_stamp: "Formatted at 2026-09-26 16:37:41 on dist-test-slave-whjp"
server_key: "94b09cfe6b9c6bbec7832cd308890846"
server_key_iv: "51302eba0922f86be38e6245272cb906"
server_key_version: "encryptionkey@0"
I20260926 16:37:41.773856 16484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "4edbc8b888144b49bc15da38a29681bd"
format_stamp: "Formatted at 2026-09-26 16:37:41 on dist-test-slave-whjp"
server_key: "94b09cfe6b9c6bbec7832cd308890846"
server_key_iv: "51302eba0922f86be38e6245272cb906"
server_key_version: "encryptionkey@0"
I20260926 16:37:41.778792 16484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:37:41.782106 16499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:41.783341 16484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:37:41.783480 16484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "4edbc8b888144b49bc15da38a29681bd"
format_stamp: "Formatted at 2026-09-26 16:37:41 on dist-test-slave-whjp"
server_key: "94b09cfe6b9c6bbec7832cd308890846"
server_key_iv: "51302eba0922f86be38e6245272cb906"
server_key_version: "encryptionkey@0"
I20260926 16:37:41.783653 16484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:41.797710 16484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:41.802383 16484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:41.802675 16484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:41.803701 16484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:41.805032 16484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:41.805107 16484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:41.805226 16484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:41.805285 16484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:41.834359 16484 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:37555
I20260926 16:37:41.834395 16611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:37555 every 8 connection(s)
I20260926 16:37:41.835551 16484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:41.842474 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16484
I20260926 16:37:41.842624 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:41.843042 10680 external_mini_cluster.cc:1428] Setting key be9ab6d441b64194eda906f922a3226c
Sep 26 16:37:41 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440661, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:41.854111 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:41.837941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:47017 (local address 127.10.110.62:33933)
0926 16:37:41.838092 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:41.838097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:41.839265 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 16:37:41.841055 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:41.841061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:41.841140 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:41.841223 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:41.842598 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.843377 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.845131 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.845508 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.845609 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:41.848864 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:41.848869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:41.848872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:41.848901 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:41.851192 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:41.851959 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:41.851963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:41.851965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:41.852110 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:41.852549 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:41.852553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:41.852555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:41.852865 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:41.852972 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:41.853688 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 16:37:41.853879 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:37:41.855319 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:41.838396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33933 (local address 127.10.110.2:47017)
0926 16:37:41.839076 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:41.839113 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:41.840753 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:41.841447 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:41.841455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:41.841914 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:41.842438 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:41.842454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.843518 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.843521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:41.845010 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:41.845022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:41.845699 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:41.845707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:41.845962 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:41.846605 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:41.846627 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:41.848678 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:41.851313 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:41.851322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:41.851336 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:41.851815 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:41.852220 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:41.852225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:41.852229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:41.852443 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:41.852973 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:41.852980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:41.853347 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 16:37:41.853605 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 16:37:41.854052 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
I20260926 16:37:41.856635 16612 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33933
I20260926 16:37:41.856909 16612 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:41.857770 16612 heartbeater.cc:511] Master 127.10.110.62:33933 requested a full tablet report, sending...
I20260926 16:37:41.859175 16289 ts_manager.cc:194] Registered new tserver with Master: 4edbc8b888144b49bc15da38a29681bd (127.10.110.2:37555)
I20260926 16:37:41.860013 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:47017
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:37:41.885057 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33933
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:42.185741 16617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:42.185995 16617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:42.186110 16617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:42.193146 16617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:42.193261 16617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:42.193413 16617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:42.202651 16617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:42.203881 16617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:42.205106 16617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:42.216838 16622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:42.216876 16623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:42.216898 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:42.217865 16617 server_base.cc:1034] running on GCE node
I20260926 16:37:42.218415 16617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:42.219307 16617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:42.220534 16617 hybrid_clock.cc:648] HybridClock initialized: now 1790440662220485 us; error 41 us; skew 500 ppm
I20260926 16:37:42.220813 16617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.223570 16617 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:37:42.224849 16617 webserver.cc:460] Webserver started at http://127.10.110.3:45225/ using document root <none> and password file <none>
I20260926 16:37:42.225452 16617 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:42.225544 16617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:42.225824 16617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:42.228646 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "1a9f0b7710d8417e94f5d4309b3b0d3d"
format_stamp: "Formatted at 2026-09-26 16:37:42 on dist-test-slave-whjp"
server_key: "7b11edf3c2f97456098d4c15400d94ee"
server_key_iv: "692a46f8dd4e4ceab3c138e6601bcb63"
server_key_version: "encryptionkey@0"
I20260926 16:37:42.229347 16617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "1a9f0b7710d8417e94f5d4309b3b0d3d"
format_stamp: "Formatted at 2026-09-26 16:37:42 on dist-test-slave-whjp"
server_key: "7b11edf3c2f97456098d4c15400d94ee"
server_key_iv: "692a46f8dd4e4ceab3c138e6601bcb63"
server_key_version: "encryptionkey@0"
I20260926 16:37:42.234315 16617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:37:42.237569 16632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:42.238873 16617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:37:42.239004 16617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "1a9f0b7710d8417e94f5d4309b3b0d3d"
format_stamp: "Formatted at 2026-09-26 16:37:42 on dist-test-slave-whjp"
server_key: "7b11edf3c2f97456098d4c15400d94ee"
server_key_iv: "692a46f8dd4e4ceab3c138e6601bcb63"
server_key_version: "encryptionkey@0"
I20260926 16:37:42.239166 16617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:42.259583 16617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:42.263857 16617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:42.264086 16617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:42.264957 16617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:42.266157 16617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:42.266237 16617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:42.266327 16617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:42.266366 16617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:42.295113 16617 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38293
I20260926 16:37:42.295154 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38293 every 8 connection(s)
I20260926 16:37:42.296312 16617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:42.300858 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16617
I20260926 16:37:42.301002 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:42.301421 10680 external_mini_cluster.cc:1428] Setting key 513bc7d9e8d35e7c23a7663f6a27bec4
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:37:42.314211 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.298684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:49913 (local address 127.10.110.62:33933)
0926 16:37:42.298858 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.298863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.300049 (+ 1186us) server_negotiation.cc:407] Connection header received
0926 16:37:42.301874 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.301880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.301959 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.302044 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.303447 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.304223 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.305664 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.306048 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.306149 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.309243 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.309247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.309251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.309281 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.311486 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.312155 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.312160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.312163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.312272 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.312644 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.312647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.312649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.312914 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.313036 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.313669 (+ 633us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.313884 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:37:42.315253 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.299168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33933 (local address 127.10.110.3:49913)
0926 16:37:42.299910 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:42.299949 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:42.301624 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:42.302282 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:42.302290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:42.302766 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:42.303294 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:42.303311 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.304356 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.304360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:42.305534 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:42.305547 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.306259 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.306266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:42.306481 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.307034 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:42.307058 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:42.309050 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:42.311607 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:42.311615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:42.311632 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:42.312052 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:42.312366 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:42.312371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:42.312374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:42.312536 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:42.313030 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:42.313037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:42.313350 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 16:37:42.313570 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 16:37:42.314041 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":267,"threads_started":1}
I20260926 16:37:42.316491 16745 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33933
I20260926 16:37:42.316754 16745 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:42.317389 16745 heartbeater.cc:511] Master 127.10.110.62:33933 requested a full tablet report, sending...
I20260926 16:37:42.318646 16289 ts_manager.cc:194] Registered new tserver with Master: 1a9f0b7710d8417e94f5d4309b3b0d3d (127.10.110.3:38293)
I20260926 16:37:42.319387 16289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:49913
I20260926 16:37:42.327997 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/test-admin.keytab.
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.352972 10680 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 16:37:42.358251 16752 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:42.355582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.10.110.1:46371)
0926 16:37:42.356399 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.356407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.356438 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:37:42.356507 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.356514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.356757 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.356919 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.357846 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45598: BlockingRecv error: recv got EOF from 127.0.0.1:45598 (error 108)
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":479,"threads_started":1}
W20260926 16:37:42.358319 10680 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.359875 10680 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.361091 10680 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:42.377049 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.362591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.10.110.1:46371)
0926 16:37:42.362772 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.362778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.362933 (+ 155us) server_negotiation.cc:407] Connection header received
0926 16:37:42.363095 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.363099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.363157 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.363244 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.364192 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.365167 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.366355 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.366901 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.367016 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.370701 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.370711 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.370734 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.370777 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.373785 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.374579 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.374590 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.374593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.374753 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.375181 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.375188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.375191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.375720 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.376060 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.376434 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.376706 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:37:42.384296 16752 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:42.381994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.10.110.1:46371)
0926 16:37:42.382176 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.382180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.383157 (+ 977us) server_negotiation.cc:407] Connection header received
0926 16:37:42.383265 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.383269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.383339 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.383421 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.384072 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45612: BlockingRecv error: recv got EOF from 127.0.0.1:45612 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 16:37:42.384500 10680 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.385972 10680 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.387246 10680 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:42.399994 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.388469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.10.110.1:46371)
0926 16:37:42.388623 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.388629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.388962 (+ 333us) server_negotiation.cc:407] Connection header received
0926 16:37:42.389104 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.389108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.389158 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.389238 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.390151 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.390862 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.392045 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.392466 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.392571 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.395660 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.395665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.395668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.395696 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.397812 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.398352 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.398358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.398360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.398467 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.398789 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.398794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.398797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.399031 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.399252 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.399519 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.399702 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 16:37:42.403475 16752 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:42.402130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.10.110.1:46371)
0926 16:37:42.402294 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.402299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.402470 (+ 171us) server_negotiation.cc:407] Connection header received
0926 16:37:42.402604 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.402608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.402674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.402760 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.403291 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45628: BlockingRecv error: recv got EOF from 127.0.0.1:45628 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:37:42.403658 10680 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440615325743-10680-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.404995 10680 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:42.406097 10680 init.cc:295] Successfully reacquired a new kerberos TGT
I20260926 16:37:42.408368 16478 heartbeater.cc:503] Master 127.10.110.62:33933 was elected leader, sending a full tablet report...
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:37:42.418841 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.407318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.10.110.1:46371)
0926 16:37:42.407490 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.407495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.407598 (+ 103us) server_negotiation.cc:407] Connection header received
0926 16:37:42.407700 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.407703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.407755 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.407825 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.408691 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.409471 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.410697 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.411075 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.411176 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.414374 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.414378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.414381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.414409 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.416614 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.417145 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.417150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.417153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.417295 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.417604 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.417608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.417610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.417863 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.418128 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.418388 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.418581 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:37:42.420204 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16350
I20260926 16:37:42.431010 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.420856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.10.110.1:46371)
0926 16:37:42.420998 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.421002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.421244 (+ 242us) server_negotiation.cc:407] Connection header received
0926 16:37:42.421351 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.421355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.421409 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.421493 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.422254 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.422945 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.424144 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.424569 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.424676 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.426544 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.426549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.426552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.426581 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.429017 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.429549 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.429552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.429555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.429671 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.429967 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.429971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.429973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.430232 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.430403 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.430635 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.430771 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:37:42.431470 16473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:42.511914 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16350
I20260926 16:37:42.524911 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16484
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:37:42.539613 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.525755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.10.110.2:37555)
0926 16:37:42.526290 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.526297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.526322 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:37:42.526380 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.526386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.526603 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.526735 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.527660 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.528729 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.529962 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.530526 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.530629 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.534268 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.534281 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.534295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.534330 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.537014 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.537805 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.537812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.537814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.537934 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.538268 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.538272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.538274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.538592 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.538823 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.539051 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.539316 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":270,"threads_started":1}
I20260926 16:37:42.540855 16607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:42.621256 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16484
I20260926 16:37:42.633574 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16617
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:37:42.649737 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:42.634670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.10.110.3:38293)
0926 16:37:42.635297 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:42.635305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:42.635334 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:37:42.635393 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:42.635400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:42.635596 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:42.635747 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:42.636909 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.638006 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.639349 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:42.640039 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:42.640177 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:42.644362 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:42.644373 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:42.644387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:42.644426 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:42.647086 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.647812 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.647819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.647821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.647979 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:42.648311 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:42.648316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:42.648319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:42.648656 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:42.648944 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:42.649199 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 16:37:42.649443 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
I20260926 16:37:42.650980 16740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:42.732287 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16617
I20260926 16:37:42.744978 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16257
I20260926 16:37:42.746089 16319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:42.847944 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16257
2026-09-26T16:37:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3669 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16772](info): set up 2 sockets
Sep 26 16:37:42 dist-test-slave-whjp krb5kdc[16772](info): commencing operation
krb5kdc: starting...
W20260926 16:37:43.849531 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.010s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:43 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab.
Entry for principal oryx/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab.
I20260926 16:37:44.034536 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34111
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab
--principal=oryx/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:44.341913 16785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:44.342190 16785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:44.342319 16785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:44.349350 16785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:44.349443 16785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:44.349514 16785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:44.349584 16785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:44.349641 16785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:44.359248 16785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab
--principal=oryx/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:44.360517 16785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:44.361781 16785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:44.373744 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:44.374184 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:44.374275 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:44.374781 16785 server_base.cc:1034] running on GCE node
I20260926 16:37:44.375653 16785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:44.377038 16785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:44.378227 16785 hybrid_clock.cc:648] HybridClock initialized: now 1790440664378182 us; error 41 us; skew 500 ppm
I20260926 16:37:44.378535 16785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:44 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440664, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:44.381726 16785 init.cc:377] Logged in from keytab as oryx/127.10.110.62@KRBTEST.COM (short username oryx)
I20260926 16:37:44.383368 16785 webserver.cc:460] Webserver started at http://127.10.110.62:36831/ using document root <none> and password file <none>
I20260926 16:37:44.384133 16785 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:44.384254 16785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:44.384527 16785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:44.387605 16785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "391f2ba7fa4c47f3bad51f6035a9113e"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "6fc3d82cb23677b9c8f40fd620b0833e"
server_key_iv: "6b50ed37a251f16c5960ca14bb60e9d3"
server_key_version: "encryptionkey@0"
I20260926 16:37:44.388432 16785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "391f2ba7fa4c47f3bad51f6035a9113e"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "6fc3d82cb23677b9c8f40fd620b0833e"
server_key_iv: "6b50ed37a251f16c5960ca14bb60e9d3"
server_key_version: "encryptionkey@0"
I20260926 16:37:44.394711 16785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:37:44.398806 16800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:44.400519 16785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:37:44.400712 16785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "391f2ba7fa4c47f3bad51f6035a9113e"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "6fc3d82cb23677b9c8f40fd620b0833e"
server_key_iv: "6b50ed37a251f16c5960ca14bb60e9d3"
server_key_version: "encryptionkey@0"
I20260926 16:37:44.400916 16785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:44.416214 16785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:44.420475 16785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:44.420714 16785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:44.438361 16785 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:34111
I20260926 16:37:44.438391 16851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:34111 every 8 connection(s)
I20260926 16:37:44.439746 16785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:44.442435 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16785
I20260926 16:37:44.442646 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:44.443430 10680 external_mini_cluster.cc:1428] Setting key 45e9f206981c5d93e2de25fc0a9aa914
I20260926 16:37:44.444190 16852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:44.456444 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Bootstrap starting.
Sep 26 16:37:44 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:44.459517 16852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:44.460644 16852 log.cc:824] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:44.463429 16852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: No bootstrap required, opened a new log
I20260926 16:37:44.465734 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:44.445579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.10.110.62:34111)
0926 16:37:44.446336 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:44.446348 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:44.446396 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:37:44.448868 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:44.448893 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:44.449354 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:44.449734 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:44.451248 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:44.452561 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:44.453803 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:44.454521 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:44.454691 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:44.458472 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:44.458480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:44.458495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:44.458536 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:44.461434 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:44.462081 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:44.462088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:44.462095 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:44.462271 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:44.462648 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:44.462652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:44.462654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:44.463165 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:44.463459 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:44.463934 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 16:37:44.464404 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":326,"threads_started":1}
I20260926 16:37:44.471930 16852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } }
I20260926 16:37:44.472272 16852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:44.472364 16852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391f2ba7fa4c47f3bad51f6035a9113e, State: Initialized, Role: FOLLOWER
I20260926 16:37:44.472977 16852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } }
I20260926 16:37:44.473183 16852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:44.473277 16852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:44.473402 16852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:44.475488 16852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } }
I20260926 16:37:44.476004 16852 leader_election.cc:302] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391f2ba7fa4c47f3bad51f6035a9113e; no voters:
I20260926 16:37:44.476539 16852 leader_election.cc:288] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:44.476665 16857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:44.477468 16857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 LEADER]: Becoming Leader. State: Replica: 391f2ba7fa4c47f3bad51f6035a9113e, State: Running, Role: LEADER
I20260926 16:37:44.477981 16857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } }
I20260926 16:37:44.478598 16852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:44.480825 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } }
I20260926 16:37:44.480851 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391f2ba7fa4c47f3bad51f6035a9113e. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } }
I20260926 16:37:44.481287 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:44.481844 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:44.482029 16864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:44.491269 16864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:44.499485 16864 catalog_manager.cc:1347] Generated new cluster ID: d8939649e04d429499593967064a2623
I20260926 16:37:44.499591 16864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:44.522516 16864 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:44.523816 16864 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:44.531728 16864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Generated new TSK 0
I20260926 16:37:44.532533 16864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab.
Entry for principal oryx/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab.
I20260926 16:37:44.576467 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34111
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:44.886473 16878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:44.886751 16878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:44.886878 16878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:44.894214 16878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:44.894328 16878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:44.894488 16878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:44.904291 16878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:44.905718 16878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:44.907256 16878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:44.919986 16883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:44.920068 16884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:44.920042 16886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:44.921471 16878 server_base.cc:1034] running on GCE node
I20260926 16:37:44.922070 16878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:44.923013 16878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:44.924228 16878 hybrid_clock.cc:648] HybridClock initialized: now 1790440664924190 us; error 40 us; skew 500 ppm
I20260926 16:37:44.924535 16878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:44 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440664, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:44.927628 16878 init.cc:377] Logged in from keytab as oryx/127.10.110.1@KRBTEST.COM (short username oryx)
I20260926 16:37:44.929088 16878 webserver.cc:460] Webserver started at http://127.10.110.1:45157/ using document root <none> and password file <none>
I20260926 16:37:44.929746 16878 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:44.929842 16878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:44.930094 16878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:44.932919 16878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "72159ff4d4ec48ffaa44b29765a6d720"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "8f614ee3dcc6111bd8b227fcdc4056d5"
server_key_iv: "de2633e76d7500849ee3095dda240a5d"
server_key_version: "encryptionkey@0"
I20260926 16:37:44.933636 16878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "72159ff4d4ec48ffaa44b29765a6d720"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "8f614ee3dcc6111bd8b227fcdc4056d5"
server_key_iv: "de2633e76d7500849ee3095dda240a5d"
server_key_version: "encryptionkey@0"
I20260926 16:37:44.938567 16878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:37:44.944458 16893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:44.950522 16878 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260926 16:37:44.950790 16878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "72159ff4d4ec48ffaa44b29765a6d720"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "8f614ee3dcc6111bd8b227fcdc4056d5"
server_key_iv: "de2633e76d7500849ee3095dda240a5d"
server_key_version: "encryptionkey@0"
I20260926 16:37:44.951032 16878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:44.990490 16878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:44.994518 16878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:44.994755 16878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:44.995770 16878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:44.997133 16878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:44.997219 16878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:44.997351 16878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:44.997416 16878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.026363 16878 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:36447
I20260926 16:37:45.026420 17005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:36447 every 8 connection(s)
I20260926 16:37:45.027621 16878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:45.035885 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 16878
I20260926 16:37:45.036062 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:45.036665 10680 external_mini_cluster.cc:1428] Setting key a54b64c9f6ec3b31f2980dd6f66a7cff
Sep 26 16:37:45 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440664, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:45.046489 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:45.030028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:53507 (local address 127.10.110.62:34111)
0926 16:37:45.030428 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:45.030433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:45.031376 (+ 943us) server_negotiation.cc:407] Connection header received
0926 16:37:45.033140 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:45.033145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:45.033229 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:45.033318 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:45.034761 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.035476 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.037132 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.037538 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.037661 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:45.041300 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:45.041305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:45.041308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:45.041340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:45.043622 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:45.044317 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:45.044323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:45.044326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:45.044463 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:45.044803 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:45.044808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:45.044810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:45.045161 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:45.045291 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:45.045945 (+ 654us) server_negotiation.cc:299] Negotiation successful
0926 16:37:45.046144 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":199,"threads_started":1}
I20260926 16:37:45.047214 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:45.030593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:53507)
0926 16:37:45.031223 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:45.031265 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:45.032889 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:45.033552 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:45.033560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:45.034054 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:45.034599 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:45.034620 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.035638 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.035642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:45.036994 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:45.037005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.037722 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.037727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:45.037915 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:45.038607 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:45.038635 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:45.041101 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:45.043740 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:45.043749 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:45.043765 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:45.044204 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:45.044548 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:45.044551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:45.044553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:45.044707 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:45.045308 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:45.045314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:45.045601 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 16:37:45.045813 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 16:37:45.046179 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":240,"threads_started":1}
I20260926 16:37:45.048615 17006 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34111
I20260926 16:37:45.048929 17006 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:45.049685 17006 heartbeater.cc:511] Master 127.10.110.62:34111 requested a full tablet report, sending...
I20260926 16:37:45.051690 16817 ts_manager.cc:194] Registered new tserver with Master: 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1:36447)
I20260926 16:37:45.053166 16817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:53507
WARNING: no policy specified for oryx/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab.
Entry for principal oryx/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab.
I20260926 16:37:45.079402 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34111
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:45.389837 17012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:45.390129 17012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:45.390261 17012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:45.397280 17012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:45.397398 17012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:45.397543 17012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:45.406997 17012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:45.408243 17012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:45.409510 17012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:45.421649 17018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:45.421712 17017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:45.421731 17020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:45.422407 17012 server_base.cc:1034] running on GCE node
I20260926 16:37:45.422950 17012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:45.423781 17012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:45.425012 17012 hybrid_clock.cc:648] HybridClock initialized: now 1790440665424964 us; error 39 us; skew 500 ppm
I20260926 16:37:45.425297 17012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:45 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440665, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:45.428191 17012 init.cc:377] Logged in from keytab as oryx/127.10.110.2@KRBTEST.COM (short username oryx)
I20260926 16:37:45.429477 17012 webserver.cc:460] Webserver started at http://127.10.110.2:33937/ using document root <none> and password file <none>
I20260926 16:37:45.430117 17012 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:45.430215 17012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:45.430447 17012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:45.433238 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "62f56de115b446d1b5ee5054e0b038bd"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "03eb554c566dc034a054e6a60b6f4cc4"
server_key_iv: "c6b140132906ab591ce16eace9ee93fa"
server_key_version: "encryptionkey@0"
I20260926 16:37:45.433957 17012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "62f56de115b446d1b5ee5054e0b038bd"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "03eb554c566dc034a054e6a60b6f4cc4"
server_key_iv: "c6b140132906ab591ce16eace9ee93fa"
server_key_version: "encryptionkey@0"
I20260926 16:37:45.439018 17012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:37:45.443145 17027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.444685 17012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:37:45.444849 17012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "62f56de115b446d1b5ee5054e0b038bd"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "03eb554c566dc034a054e6a60b6f4cc4"
server_key_iv: "c6b140132906ab591ce16eace9ee93fa"
server_key_version: "encryptionkey@0"
I20260926 16:37:45.445063 17012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:45.463562 17012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:45.467689 17012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:45.467991 17012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:45.468902 17012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:45.470191 17012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:45.470268 17012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.470346 17012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:45.470391 17012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.498517 17012 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:46117
I20260926 16:37:45.498555 17139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:46117 every 8 connection(s)
I20260926 16:37:45.499703 17012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:45.507020 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17012
I20260926 16:37:45.507167 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:45.507580 10680 external_mini_cluster.cc:1428] Setting key 29c17f667c47ea1e8a7ecc8c214566ee
Sep 26 16:37:45 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440665, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:45.518491 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:45.502156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:57369 (local address 127.10.110.62:34111)
0926 16:37:45.502362 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:45.502368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:45.503410 (+ 1042us) server_negotiation.cc:407] Connection header received
0926 16:37:45.505163 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:45.505167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:45.505235 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:45.505328 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:45.506820 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.507581 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.509230 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.509653 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.509763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:45.513193 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:45.513198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:45.513201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:45.513233 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:45.515755 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:45.516431 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:45.516435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:45.516437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:45.516552 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:45.516942 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:45.516946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:45.516948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:45.517186 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:45.517310 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:45.517958 (+ 648us) server_negotiation.cc:299] Negotiation successful
0926 16:37:45.518169 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:37:45.519381 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:45.502605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:57369)
0926 16:37:45.503257 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:45.503298 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:45.504941 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:45.505544 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:45.505552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:45.506059 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:45.506604 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:45.506650 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.507743 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.507749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:45.509094 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:45.509106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.509882 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.509890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:45.510100 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:45.510689 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:45.510710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:45.512973 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:45.515892 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:45.515901 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:45.515918 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:45.516313 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:45.516648 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:45.516652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:45.516656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:45.516835 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:45.517306 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:45.517313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:45.517621 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 16:37:45.517839 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 16:37:45.518337 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":254,"threads_started":1}
I20260926 16:37:45.520709 17140 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34111
I20260926 16:37:45.521034 17140 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:45.521920 17140 heartbeater.cc:511] Master 127.10.110.62:34111 requested a full tablet report, sending...
I20260926 16:37:45.523454 16817 ts_manager.cc:194] Registered new tserver with Master: 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:45.524251 16817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:57369
WARNING: no policy specified for oryx/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab.
Entry for principal oryx/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab.
I20260926 16:37:45.551204 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34111
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:45.871618 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:45.871901 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:45.871994 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:45.879048 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:45.879148 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:45.879271 17145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:45.888625 17145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:45.889773 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:45.890990 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:45.903077 17151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:45.903151 17150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:45.903119 17153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:45.903908 17145 server_base.cc:1034] running on GCE node
I20260926 16:37:45.904618 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:45.905557 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:45.906776 17145 hybrid_clock.cc:648] HybridClock initialized: now 1790440665906722 us; error 48 us; skew 500 ppm
I20260926 16:37:45.907078 17145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:45 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440665, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:45.913057 17145 init.cc:377] Logged in from keytab as oryx/127.10.110.3@KRBTEST.COM (short username oryx)
I20260926 16:37:45.914423 17145 webserver.cc:460] Webserver started at http://127.10.110.3:37599/ using document root <none> and password file <none>
I20260926 16:37:45.915071 17145 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:45.915181 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:45.915458 17145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:45.918418 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "c63c9ec9560440b1bb66ea08b359a00e"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "c606ace2444509f60fc1de1d2402f5f7"
server_key_iv: "ca58ff11470deb1e2a037e8249d45a54"
server_key_version: "encryptionkey@0"
I20260926 16:37:45.919122 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "c63c9ec9560440b1bb66ea08b359a00e"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "c606ace2444509f60fc1de1d2402f5f7"
server_key_iv: "ca58ff11470deb1e2a037e8249d45a54"
server_key_version: "encryptionkey@0"
I20260926 16:37:45.923974 17145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:37:45.927093 17160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.928403 17145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:45.928537 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "c63c9ec9560440b1bb66ea08b359a00e"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "c606ace2444509f60fc1de1d2402f5f7"
server_key_iv: "ca58ff11470deb1e2a037e8249d45a54"
server_key_version: "encryptionkey@0"
I20260926 16:37:45.928695 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:45.939222 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:45.942662 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:45.942912 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:45.943742 17145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:45.944968 17145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:45.945042 17145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.945155 17145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:45.945206 17145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:45.974890 17145 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:44193
I20260926 16:37:45.974936 17272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:44193 every 8 connection(s)
I20260926 16:37:45.976119 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:45.979192 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17145
I20260926 16:37:45.979350 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:45.979749 10680 external_mini_cluster.cc:1428] Setting key ec2c86c86e6f23dc25ebf4370e28dfdd
Sep 26 16:37:45 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440665, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:45.994768 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:45.978457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:49563 (local address 127.10.110.62:34111)
0926 16:37:45.978617 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:45.978621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:45.979886 (+ 1265us) server_negotiation.cc:407] Connection header received
0926 16:37:45.981910 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:45.981915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:45.981994 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:45.982098 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:45.983505 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.984384 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.985845 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.986212 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.986297 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:45.989254 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:45.989260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:45.989264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:45.989296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:45.991692 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:45.992440 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:45.992444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:45.992446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:45.992558 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:45.992968 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:45.992973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:45.992976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:45.993298 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:45.993453 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:45.994279 (+ 826us) server_negotiation.cc:299] Negotiation successful
0926 16:37:45.994456 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:37:45.995781 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:45.978960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:49563)
0926 16:37:45.979689 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:45.979733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:45.981668 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:45.982278 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:45.982288 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:45.982783 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:45.983358 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:45.983377 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.984502 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.984505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:45.985732 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:45.985744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:45.986362 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:45.986367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:45.986544 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:45.987086 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:45.987107 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:45.989065 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:45.991821 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:45.991861 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:45.991880 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:45.992338 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:45.992653 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:45.992656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:45.992658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:45.992852 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:45.993443 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:45.993451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:45.993889 (+ 438us) client_negotiation.cc:769] Sending connection context
0926 16:37:45.994176 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 16:37:45.994696 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":300,"threads_started":1}
I20260926 16:37:45.997004 17273 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34111
I20260926 16:37:45.997296 17273 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:45.997963 17273 heartbeater.cc:511] Master 127.10.110.62:34111 requested a full tablet report, sending...
I20260926 16:37:45.999323 16817 ts_manager.cc:194] Registered new tserver with Master: c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193)
I20260926 16:37:46.000208 16817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:49563
I20260926 16:37:46.006901 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.110.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.61.keytab.
Entry for principal oryx/127.10.110.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.61.keytab.
I20260926 16:37:46.048056 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:44281
--webserver_interface=127.10.110.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.61.keytab
--principal=oryx/127.10.110.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34111,127.10.110.61:44281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:37:46.056380 17006 heartbeater.cc:503] Master 127.10.110.62:34111 was elected leader, sending a full tablet report...
W20260926 16:37:46.355423 17278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:46.355682 17278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:46.355808 17278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:46.362871 17278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:46.362974 17278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:46.363029 17278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:46.363106 17278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:46.363154 17278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:46.372877 17278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34111,127.10.110.61:44281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.61.keytab
--principal=oryx/127.10.110.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:44281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.110.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
W20260926 16:37:46.373251 17278 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.110.62:34111,127.10.110.61:44281'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 16:37:46.374498 17278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:46.376097 17278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:46.387585 17283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:46.387523 17284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:46.387562 17286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:46.388238 17278 server_base.cc:1034] running on GCE node
I20260926 16:37:46.388928 17278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:46.390198 17278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:46.391426 17278 hybrid_clock.cc:648] HybridClock initialized: now 1790440666391355 us; error 46 us; skew 500 ppm
I20260926 16:37:46.391728 17278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:46.394550 17278 init.cc:377] Logged in from keytab as oryx/127.10.110.61@KRBTEST.COM (short username oryx)
I20260926 16:37:46.395892 17278 webserver.cc:460] Webserver started at http://127.10.110.61:44541/ using document root <none> and password file <none>
I20260926 16:37:46.396561 17278 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:46.396653 17278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:46.396932 17278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:46.399657 17278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/instance:
uuid: "2eca9fd88c084e51844c3d553f2cb5c1"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "b87e1e423f812665fd70c6bf9a1a0b53"
server_key_iv: "a964135eb894822dd713135677b7e4e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.400414 17278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal/instance:
uuid: "2eca9fd88c084e51844c3d553f2cb5c1"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "b87e1e423f812665fd70c6bf9a1a0b53"
server_key_iv: "a964135eb894822dd713135677b7e4e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.405140 17278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:37:46.408290 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:46.409492 17278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 16:37:46.409636 17278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
uuid: "2eca9fd88c084e51844c3d553f2cb5c1"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "b87e1e423f812665fd70c6bf9a1a0b53"
server_key_iv: "a964135eb894822dd713135677b7e4e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.409814 17278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:46.437197 17278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:46.441223 17278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:46.441473 17278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:46.457552 17278 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.61:44281
I20260926 16:37:46.457571 17344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.61:44281 every 8 connection(s)
I20260926 16:37:46.458921 17278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
I20260926 16:37:46.462476 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:46.465312 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17278
I20260926 16:37:46.469851 17345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.61@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:46.485620 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.472179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.10.110.62:34111)
0926 16:37:46.472374 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.472380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.473103 (+ 723us) server_negotiation.cc:407] Connection header received
0926 16:37:46.473266 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.473271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.473350 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.473440 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.474474 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.475175 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.476586 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.476981 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.477090 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.480408 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.480414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.480417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.480448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.482820 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.483521 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.483526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.483529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.483691 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.484220 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.484224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.484227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.484529 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.484680 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.485229 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.485401 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:37:46.486843 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.472230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:41832)
0926 16:37:46.472954 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:46.473002 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:46.473156 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:46.473504 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:46.473509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:46.474086 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:46.474333 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.474341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.475321 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.475325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.476458 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.476469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.477182 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.477188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.477333 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.477900 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:46.477920 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:46.480236 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:46.482921 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.482927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.482936 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.483425 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.483811 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.483816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.483820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.484099 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.484655 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:46.484662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:46.484849 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 16:37:46.485098 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 16:37:46.485588 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"spinlock_wait_cycles":21888,"thread_start_us":346,"threads_started":1}
I20260926 16:37:46.486868 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.467548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.10.110.61:44281)
0926 16:37:46.468321 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.468334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.468374 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:37:46.469953 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.469980 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.470392 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.470733 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.472265 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.473794 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.474990 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.475647 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.475803 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.479792 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.479800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.479811 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.479882 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.482513 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.483291 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.483300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.483303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.483483 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.483811 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.483816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.483819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.484490 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.484740 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.485043 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.485427 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":356,"threads_started":1}
I20260926 16:37:46.489300 17345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.61@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
I20260926 16:37:46.500435 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.489915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.10.110.61:44281)
0926 16:37:46.490049 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.490052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.490117 (+ 65us) server_negotiation.cc:407] Connection header received
0926 16:37:46.490225 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.490228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.490275 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.490342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.490884 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.491539 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.492614 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.493064 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.493156 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.495592 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.495596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.495599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.495625 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.497833 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.498591 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.498597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.498600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.498728 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.499082 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.499085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.499087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.499358 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.499454 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.499887 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.500182 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":38}
I20260926 16:37:46.500439 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.489889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43278)
0926 16:37:46.490064 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:46.490079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:46.490159 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:46.490362 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:46.490365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:46.490643 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:46.490767 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.490774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.491646 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.491649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.492512 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.492523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.493173 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.493177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.493310 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.493727 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:46.493741 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:46.495441 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:46.497974 (+ 2533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.497993 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.497996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.498478 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.498830 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.498835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.498838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.498989 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.499485 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:46.499490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:46.499632 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 16:37:46.499771 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 16:37:46.500071 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 16:37:46.504732 17345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Bootstrap starting.
I20260926 16:37:46.507613 17345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:46.508597 17345 log.cc:824] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:46.511096 17345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: No bootstrap required, opened a new log
I20260926 16:37:46.517886 17345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } }
I20260926 16:37:46.518165 17345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:46.518250 17345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eca9fd88c084e51844c3d553f2cb5c1, State: Initialized, Role: FOLLOWER
I20260926 16:37:46.518765 17345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } }
I20260926 16:37:46.519528 17351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } } }
I20260926 16:37:46.519809 17351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:37:46.520439 17345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:46.526821 17140 heartbeater.cc:503] Master 127.10.110.62:34111 was elected leader, sending a full tablet report...
W20260926 16:37:46.528791 17362 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:37:46.528901 17362 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:37:46.530821 17278 master_runner.cc:428] Detected that this master 2eca9fd88c084e51844c3d553f2cb5c1 is joining an existing cluster
I20260926 16:37:46.530925 17278 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.110.61:44281
I20260926 16:37:46.549995 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.534964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.10.110.62:34111)
0926 16:37:46.535135 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.535142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.535813 (+ 671us) server_negotiation.cc:407] Connection header received
0926 16:37:46.536104 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.536110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.536184 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.536247 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.537688 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.538714 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.540793 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.541393 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.541470 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.543608 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.543614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.543618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.543652 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.546754 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.547461 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.547467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.547472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.547616 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.548078 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.548083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.548087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.548523 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.548649 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.549390 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.549608 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:37:46.551051 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.535364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.10.110.61:44281)
0926 16:37:46.535553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.535563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.536301 (+ 738us) server_negotiation.cc:407] Connection header received
0926 16:37:46.536437 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.536442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.536486 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.536564 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.537585 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.538673 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.540592 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.541294 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.541408 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.544803 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.544809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.544817 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.544849 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.548001 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.548827 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.548832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.548835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.548974 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.549338 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.549342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.549349 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.549668 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.549842 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.550352 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.550672 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":59}
I20260926 16:37:46.551380 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.535064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:41844)
0926 16:37:46.535712 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:46.535739 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:46.535955 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:46.536453 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:46.536458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:46.537153 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:46.537544 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.537556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.538881 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.538884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.540135 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.540148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.541704 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.541709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.541934 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.542616 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:46.542638 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:46.543424 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:46.546908 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.546913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.546917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.547323 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.547755 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.547760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.547764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.547957 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.548743 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:46.548748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:46.548964 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 16:37:46.550234 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0926 16:37:46.550542 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":21,"thread_start_us":349,"threads_started":1}
I20260926 16:37:46.551863 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.535075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43282)
0926 16:37:46.536214 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:46.536228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:46.536334 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:46.536586 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:46.536590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:46.537149 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:46.537460 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.537469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.538868 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.538873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.540461 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.540475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.541677 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.541687 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.541889 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.543629 (+ 1740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:46.543649 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:46.544640 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:46.548108 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.548112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.548115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.548726 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.549100 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.549103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.549106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.549249 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.549917 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:46.549921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:46.550059 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 16:37:46.550427 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 16:37:46.550867 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":680,"thread_start_us":355,"threads_started":1}
WARNING: no policy specified for oryx/127.10.110.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.60@KRBTEST.COM" created.
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440664, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.62@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
I20260926 16:37:46.575011 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.558750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43286)
0926 16:37:46.559379 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:46.559427 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:46.559665 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:46.560388 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:46.560395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:46.561264 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:46.561545 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.561558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.562951 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.562955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.563996 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.564008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.565053 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.565060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.565247 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.565876 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:46.565896 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:46.568524 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:46.572151 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.572162 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.572170 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.572797 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.573184 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.573188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.573192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.573339 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.573928 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:46.573936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:46.574113 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 16:37:46.574309 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 16:37:46.574599 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":548,"threads_started":1}
I20260926 16:37:46.575001 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.559092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.10.110.61:44281)
0926 16:37:46.559255 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.559261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.559506 (+ 245us) server_negotiation.cc:407] Connection header received
0926 16:37:46.559923 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.559929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.560037 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.560105 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.561721 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.562835 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.564120 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.564887 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.565007 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.568719 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.568725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.568730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.568763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.572010 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.572913 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.572919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.572922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.573078 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.573443 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.573448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.573452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.573787 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.573930 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.574421 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.574638 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:37:46.577468 16817 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.110.61:44281: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1"
member_type: NON_VOTER
last_known_addr {
host: "127.10.110.61"
port: 44281
}
attrs {
promote: true
}
}
dest_uuid: "391f2ba7fa4c47f3bad51f6035a9113e"
cas_config_opid_index: -1
I20260926 16:37:46.578682 16817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } }
I20260926 16:37:46.583786 17375 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2eca9fd88c084e51844c3d553f2cb5c1 (127.10.110.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } }
I20260926 16:37:46.586156 17320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:46.588631 17375 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.110.61:44281
I20260926 16:37:46.588995 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } }
I20260926 16:37:46.589301 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:46.589383 17278 master.cc:537] Master@127.10.110.61:44281 shutting down...
I20260926 16:37:46.590314 17320 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Refusing update from remote peer 391f2ba7fa4c47f3bad51f6035a9113e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.60.keytab.
Entry for principal oryx/127.10.110.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.60.keytab.
I20260926 16:37:46.592038 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:42269
--webserver_interface=127.10.110.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.60.keytab
--principal=oryx/127.10.110.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:37:46.593257 17374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:37:46.595765 16802 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 16:37:46.601572 17278 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:46.602039 17278 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:46.602293 17278 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: stopping tablet replica
I20260926 16:37:46.609180 17278 master.cc:552] Master@127.10.110.61:44281 shutdown complete.
I20260926 16:37:46.609355 17278 master_runner.cc:305] Clearing existing system tablet
I20260926 16:37:46.609809 17278 fs_manager.cc:359] Metadata directory not provided
W20260926 16:37:46.609908 16802 consensus_peers.cc:487] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e -> Peer 2eca9fd88c084e51844c3d553f2cb5c1 (127.10.110.61:44281): Couldn't send request to peer 2eca9fd88c084e51844c3d553f2cb5c1. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260926 16:37:46.609961 17278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:46.612910 17278 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 16:37:46.614603 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:46.614948 17278 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:46.615356 17278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 16:37:46.615522 17278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
uuid: "2eca9fd88c084e51844c3d553f2cb5c1"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "b87e1e423f812665fd70c6bf9a1a0b53"
server_key_iv: "a964135eb894822dd713135677b7e4e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.620438 17278 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:46.627451 17278 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:37:46.627724 17278 log.cc:1196] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 16:37:46.628268 17278 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Deleting consensus metadata
I20260926 16:37:46.628558 17278 master_runner.cc:315] Copying system tablet from 127.10.110.62:34111
I20260926 16:37:46.631021 17278 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: tablet copy: Beginning tablet copy session from remote peer at address 127.10.110.62:34111
I20260926 16:37:46.650257 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.631891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:41852)
0926 16:37:46.636257 (+ 4366us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:46.636279 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:46.636449 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:46.636864 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:46.636869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:46.637373 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:46.637644 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.637657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.639095 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.639100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.640651 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:46.640667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.641637 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.641643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:46.641828 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.642504 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:46.642527 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:46.643572 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:46.647353 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.647359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.647363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.647898 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.648296 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:46.648301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:46.648305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:46.648467 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:46.649111 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:46.649120 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:46.649296 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 16:37:46.649452 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 16:37:46.649775 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4216,"thread_start_us":4106,"threads_started":1}
I20260926 16:37:46.650332 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:46.631997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.10.110.62:34111)
0926 16:37:46.632190 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:46.632205 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:46.636399 (+ 4194us) server_negotiation.cc:407] Connection header received
0926 16:37:46.636609 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:46.636614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:46.636722 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:46.636829 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:46.637823 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.638937 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.640811 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:46.641455 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:46.641590 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:46.643806 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:46.643813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:46.643817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:46.643890 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:46.647214 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.648013 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.648019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.648022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.648174 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:46.648585 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:46.648590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:46.648593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:46.648988 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:46.649162 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:46.649588 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 16:37:46.650015 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:37:46.651640 16837 tablet_copy_service.cc:140] P 391f2ba7fa4c47f3bad51f6035a9113e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2eca9fd88c084e51844c3d553f2cb5c1 ({username='oryx', principal='oryx/127.10.110.61@KRBTEST.COM'} at 127.0.0.1:41852)
I20260926 16:37:46.651968 16837 tablet_copy_service.cc:161] P 391f2ba7fa4c47f3bad51f6035a9113e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2eca9fd88c084e51844c3d553f2cb5c1 at {username='oryx', principal='oryx/127.10.110.61@KRBTEST.COM'} at 127.0.0.1:41852: session id = 2eca9fd88c084e51844c3d553f2cb5c1-00000000000000000000000000000000
I20260926 16:37:46.656123 16837 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:37:46.658445 17278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:37:46.665026 17278 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: tablet copy: Starting download of 0 data blocks...
I20260926 16:37:46.665349 17278 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: tablet copy: Starting download of 1 WAL segments...
I20260926 16:37:46.668151 17278 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:37:46.672469 16837 tablet_copy_service.cc:342] P 391f2ba7fa4c47f3bad51f6035a9113e: Request end of tablet copy session 2eca9fd88c084e51844c3d553f2cb5c1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.110.61@KRBTEST.COM'} at 127.0.0.1:41852
I20260926 16:37:46.672726 16837 tablet_copy_service.cc:433] P 391f2ba7fa4c47f3bad51f6035a9113e: ending tablet copy session 2eca9fd88c084e51844c3d553f2cb5c1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2eca9fd88c084e51844c3d553f2cb5c1
I20260926 16:37:46.683382 17278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:46.686158 17387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:46.687258 17388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:46.694072 17390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:46.694314 17278 server_base.cc:1034] running on GCE node
I20260926 16:37:46.694751 17278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:46.695433 17278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:46.696688 17278 hybrid_clock.cc:648] HybridClock initialized: now 1790440666696630 us; error 31 us; skew 500 ppm
I20260926 16:37:46.696986 17278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:46.699136 17278 init.cc:377] Logged in from keytab as oryx/127.10.110.61@KRBTEST.COM (short username oryx)
I20260926 16:37:46.700604 17278 webserver.cc:460] Webserver started at http://127.10.110.61:35363/ using document root <none> and password file <none>
I20260926 16:37:46.700990 17278 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:46.701125 17278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:46.704294 17278 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:37:46.706184 17397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:46.706570 17278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 16:37:46.706740 17278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
uuid: "2eca9fd88c084e51844c3d553f2cb5c1"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "b87e1e423f812665fd70c6bf9a1a0b53"
server_key_iv: "a964135eb894822dd713135677b7e4e7"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.706965 17278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:46.721980 17278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:46.725950 17278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:46.754806 17278 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.61:44281
I20260926 16:37:46.754828 17448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.61:44281 every 8 connection(s)
I20260926 16:37:46.756248 17278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
I20260926 16:37:46.762032 17449 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:37:46.765561 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Bootstrap starting.
I20260926 16:37:46.783390 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:37:46.784004 17449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Bootstrap complete.
I20260926 16:37:46.785081 17449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } }
I20260926 16:37:46.785338 17449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2eca9fd88c084e51844c3d553f2cb5c1, State: Initialized, Role: LEARNER
I20260926 16:37:46.785571 17449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } }
I20260926 16:37:46.786398 17452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } }
I20260926 16:37:46.786643 17449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:46.786842 17452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:37:46.794986 17463 catalog_manager.cc:1259] Loaded cluster ID: d8939649e04d429499593967064a2623
I20260926 16:37:46.795095 17463 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: loading cluster ID for follower catalog manager: success
I20260926 16:37:46.798424 17463 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: acquiring CA information for follower catalog manager: success
I20260926 16:37:46.800804 17463 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:37:46.942051 17378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:46.942328 17378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:46.942453 17378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:46.949676 17378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:46.949783 17378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:46.949857 17378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:46.949945 17378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:46.950006 17378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:46.960000 17378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.60.keytab
--principal=oryx/127.10.110.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.110.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:46.961333 17378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:46.962579 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:46.973263 17471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:46.973410 17469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:46.973313 17468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:46.974437 17378 server_base.cc:1034] running on GCE node
I20260926 16:37:46.975257 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:46.978152 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:46.979447 17378 hybrid_clock.cc:648] HybridClock initialized: now 1790440666979365 us; error 74 us; skew 500 ppm
I20260926 16:37:46.979792 17378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:46 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:46.982856 17378 init.cc:377] Logged in from keytab as oryx/127.10.110.60@KRBTEST.COM (short username oryx)
I20260926 16:37:46.984504 17378 webserver.cc:460] Webserver started at http://127.10.110.60:35377/ using document root <none> and password file <none>
I20260926 16:37:46.985221 17378 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:46.985327 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:46.985637 17378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:46.988890 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/instance:
uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "58a7a5b5f974ea4bb6ca33dd61da2a29"
server_key_iv: "558e37afcdad1f139a6a4c2980c5b8aa"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.989717 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal/instance:
uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "58a7a5b5f974ea4bb6ca33dd61da2a29"
server_key_iv: "558e37afcdad1f139a6a4c2980c5b8aa"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.994807 17378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:37:46.998147 17478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:46.999464 17378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:46.999615 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "58a7a5b5f974ea4bb6ca33dd61da2a29"
server_key_iv: "558e37afcdad1f139a6a4c2980c5b8aa"
server_key_version: "encryptionkey@0"
I20260926 16:37:46.999804 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:47.002498 17273 heartbeater.cc:503] Master 127.10.110.62:34111 was elected leader, sending a full tablet report...
I20260926 16:37:47.010524 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:47.014495 17378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:47.014751 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:47.030190 17378 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.60:42269
I20260926 16:37:47.030208 17529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.60:42269 every 8 connection(s)
I20260926 16:37:47.031404 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
I20260926 16:37:47.034964 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:47.036293 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17378
I20260926 16:37:47.043233 17530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.60@KRBTEST.COM
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.60@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:47.058645 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.038453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.10.110.60:42269)
0926 16:37:47.039184 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.039198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.039236 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:37:47.040992 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.041014 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.041384 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.041712 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.043053 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.044370 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.045622 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.046382 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.046527 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.051022 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.051032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.051046 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.051137 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.054636 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.055409 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.055415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.055421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.055571 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.055971 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.055974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.055976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.056465 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.056735 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.057113 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.057483 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":319,"threads_started":1}
I20260926 16:37:47.060925 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.045433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:41866)
0926 16:37:47.046117 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.046155 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.046314 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.046704 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.046710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.047395 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.047707 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.047719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.048743 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.048749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.050131 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.050143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.051136 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.051143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.051371 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.052009 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.052031 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.054649 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.058024 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.058033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.058044 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.058635 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.059068 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.059072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.059076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.059232 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.059868 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.059876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.060156 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.060420 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.060640 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":337,"threads_started":1}
I20260926 16:37:47.061059 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.045545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.10.110.62:34111)
0926 16:37:47.045755 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.045762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.046258 (+ 496us) server_negotiation.cc:407] Connection header received
0926 16:37:47.046448 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.046453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.046527 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.046613 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.047885 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.048601 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.050264 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.050926 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.051020 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.054811 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.054818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.054822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.054861 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.057913 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.058776 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.058781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.058784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.058925 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.059359 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.059363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.059367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.059711 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.059856 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.060523 (+ 667us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.060740 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:37:47.062153 17530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.60@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
I20260926 16:37:47.073633 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.062686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43288)
0926 16:37:47.062870 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.062887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.062960 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.063519 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.063522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.063810 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.063981 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.063988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.065197 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.065201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.066089 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.066099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.066760 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.066764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.066868 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.067333 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.067352 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.069156 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.071776 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.071780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.071783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.072164 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.072477 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.072480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.072484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.072601 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.073019 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.073022 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.073132 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.073235 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.073423 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 16:37:47.073688 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.062844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.10.110.61:44281)
0926 16:37:47.063300 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.063304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.063318 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:37:47.063363 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.063366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.063419 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.063500 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.064108 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.065100 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.066224 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.066627 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.066724 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.069318 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.069322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.069326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.069369 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.071654 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.072251 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.072255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.072257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.072382 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.072696 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.072699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.072701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.072941 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.073048 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.073311 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.073462 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260926 16:37:47.074607 17530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440666, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.60@KRBTEST.COM for oryx/127.10.110.60@KRBTEST.COM
I20260926 16:37:47.085345 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.075223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.0.0.1:37746)
0926 16:37:47.075375 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.075389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.075461 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.075741 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.075746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.076080 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.076209 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.076216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.077089 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.077093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.077906 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.077916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.078558 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.078563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.078736 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.079132 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.079151 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.080966 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.083373 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.083377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.083379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.083735 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.084069 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.084072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.084074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.084234 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.084667 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.084671 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.084808 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.084912 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.085074 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 16:37:47.085417 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.075388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.10.110.60:42269)
0926 16:37:47.075509 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.075513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.075526 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:37:47.075559 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.075563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.075630 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.075695 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.076323 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.076972 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.078009 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.078390 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.078470 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.081101 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.081105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.081108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.081133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.083271 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.083859 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.083865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.083867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.083970 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.084325 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.084328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.084330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.084560 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.084694 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.084981 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.085176 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":42}
I20260926 16:37:47.089967 17530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Bootstrap starting.
I20260926 16:37:47.092944 17530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:47.093843 17530 log.cc:824] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:47.096174 17530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: No bootstrap required, opened a new log
I20260926 16:37:47.102679 17530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } }
I20260926 16:37:47.102974 17530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:47.103052 17530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1cb8fe9d814ed1a040461f1fced1e3, State: Initialized, Role: FOLLOWER
I20260926 16:37:47.103513 17530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } }
I20260926 16:37:47.104332 17537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } } }
I20260926 16:37:47.104696 17537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:37:47.105260 17530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:47.113408 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16878
W20260926 16:37:47.113534 17548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:37:47.113653 17548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:37:47.118887 17378 master_runner.cc:428] Detected that this master 7a1cb8fe9d814ed1a040461f1fced1e3 is joining an existing cluster
I20260926 16:37:47.119009 17378 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.110.60:42269
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.1@KRBTEST.COM
I20260926 16:37:47.138785 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.114394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.10.110.1:36447)
0926 16:37:47.114913 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.114922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.114950 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:37:47.115014 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.115022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.115266 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.116815 (+ 1549us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.117023 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.118116 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.119760 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.120652 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.126790 (+ 6138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.127077 (+ 287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.127089 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.127106 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.127146 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.132581 (+ 5435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.133601 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.133610 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.133614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.133777 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.134294 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.134299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.134303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.134741 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.137903 (+ 3162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.138076 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.138370 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":872,"threads_started":1}
I20260926 16:37:47.140182 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.123805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.10.110.61:44281)
0926 16:37:47.124418 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.124422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.124435 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:37:47.124478 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.124481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.124554 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.124637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.125968 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.127168 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.130542 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.131159 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.131258 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.133733 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.133740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.133746 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.133786 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.137065 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.137905 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.137911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.137914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.138056 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.138517 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.138522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.138524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.138851 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.138962 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.139609 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.139869 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519}
I20260926 16:37:47.141284 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.122772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43292)
0926 16:37:47.123387 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.123412 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.123529 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.124680 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.124685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.125397 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.125780 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.125795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.127371 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.127378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.129153 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.129169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.131540 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.131545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.131725 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.132339 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.132370 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.133512 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.137240 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.137244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.137248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.137771 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.138198 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.138207 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.138210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.138391 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.139061 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.139065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.139304 (+ 239us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.140533 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.140830 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":1966,"threads_started":1}
I20260926 16:37:47.142781 17001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:47.149950 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.124211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.10.110.60:42269)
0926 16:37:47.124403 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.124409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.124427 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:37:47.124480 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.124484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.124555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.124651 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.130483 (+ 5832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.131647 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.139027 (+ 7380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.139796 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.139927 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.143521 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.143528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.143533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.143567 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.146958 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.147769 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.147775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.147778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.147963 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.148373 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.148378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.148382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.148744 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.148880 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.149373 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.149585 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:37:47.150940 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.123471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.0.0.1:37754)
0926 16:37:47.124102 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.124119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.124221 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.129318 (+ 5097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.129323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.130014 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.130332 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.130351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.134924 (+ 4573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.134930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.136437 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.136455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.140202 (+ 3747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.140208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.140424 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.142181 (+ 1757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.142219 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.143326 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.147099 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.147105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.147109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.147630 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.148096 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.148101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.148105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.148264 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.148952 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.148957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.149099 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.150191 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.150487 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":1538,"threads_started":1}
I20260926 16:37:47.151044 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.123983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.10.110.62:34111)
0926 16:37:47.124244 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.124250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.129330 (+ 5080us) server_negotiation.cc:407] Connection header received
0926 16:37:47.129789 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.129795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.129904 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.129973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.131350 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.132487 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.134462 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.135107 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.135203 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.137974 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.137980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.137984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.138023 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.146404 (+ 8381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.148622 (+ 2218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.148629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.148634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.148813 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.149248 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.149254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.149257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.149635 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.149768 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.150423 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.150668 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260926 16:37:47.152118 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.125471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:41868)
0926 16:37:47.129242 (+ 3771us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.129260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.129624 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.130174 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.130178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.130832 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.131185 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.131196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.132706 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.132712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.134304 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.134318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.135492 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.135499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.136059 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.136644 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.136667 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.137776 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.147969 (+ 10193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.147974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.147978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.148456 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.148960 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.148965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.148969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.149132 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.149920 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.149925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.150135 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.151220 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.151535 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3643,"thread_start_us":3874,"threads_started":1}
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440664, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.62@KRBTEST.COM for oryx/127.10.110.60@KRBTEST.COM
I20260926 16:37:47.176608 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.157657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.0.0.1:37760)
0926 16:37:47.160209 (+ 2552us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.160227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.160409 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.161607 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.161613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.162064 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.162292 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.162304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.163892 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.163898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.165263 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.165277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.166272 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.166278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.166456 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.167088 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.167120 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.169821 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.173791 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.173797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.173801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.174312 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.174740 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.174746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.174749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.174901 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.175558 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.175564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.175720 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.175902 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.176180 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2429,"thread_start_us":2317,"threads_started":1}
I20260926 16:37:47.176641 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.157714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.10.110.60:42269)
0926 16:37:47.157871 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.157876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.160309 (+ 2433us) server_negotiation.cc:407] Connection header received
0926 16:37:47.160596 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.160606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.160939 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.161023 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.162515 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.163711 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.165430 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.166096 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.166218 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.170052 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.170059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.170065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.170114 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.173649 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.174466 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.174472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.174476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.174626 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.175020 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.175025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.175029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.175435 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.175577 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.176012 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.176261 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:47.178581 16817 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.110.60:42269: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
member_type: NON_VOTER
last_known_addr {
host: "127.10.110.60"
port: 42269
}
attrs {
promote: true
}
}
dest_uuid: "391f2ba7fa4c47f3bad51f6035a9113e"
cas_config_opid_index: 5
I20260926 16:37:47.179600 16817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } }
I20260926 16:37:47.195487 17505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:47.196208 17561 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.199273 17505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Refusing update from remote peer 391f2ba7fa4c47f3bad51f6035a9113e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:37:47.200501 17561 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.110.60:42269
I20260926 16:37:47.201125 17378 master.cc:537] Master@127.10.110.60:42269 shutting down...
I20260926 16:37:47.205585 17560 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:47.210183 17562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.210680 17562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:47.216578 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.206888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43296)
0926 16:37:47.207216 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.207257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.207391 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.208322 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.208327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.208351 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:47.208508 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.208517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.211420 (+ 2903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.211426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.213589 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.213604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.215643 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.215650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.215961 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.216024 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.216118 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.216273 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260926 16:37:47.217511 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.207623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.10.110.61:44281)
0926 16:37:47.207883 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.207888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.207907 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:37:47.207974 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.207978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.208061 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.208167 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:47.208701 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.209748 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.213796 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.215154 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.215291 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.215543 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.217080 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.217227 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 16:37:47.218547 17424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 LEARNER]: Refusing update from remote peer 391f2ba7fa4c47f3bad51f6035a9113e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:37:47.219389 17561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 16:37:47.222246 17424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.228056 17452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391f2ba7fa4c47f3bad51f6035a9113e. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.228307 17452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:37:47.227620 17568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: NON_VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: true } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.228998 17568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:37:47.229363 17560 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: attempting to promote NON_VOTER 2eca9fd88c084e51844c3d553f2cb5c1 to VOTER
I20260926 16:37:47.230242 17560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } }
I20260926 16:37:47.232090 17424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 LEARNER]: Refusing update from remote peer 391f2ba7fa4c47f3bad51f6035a9113e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 16:37:47.233232 17561 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 16:37:47.233592 16801 consensus_peers.cc:487] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e -> Peer 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60:42269): Couldn't send request to peer 7a1cb8fe9d814ed1a040461f1fced1e3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:37:47.235224 17563 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2eca9fd88c084e51844c3d553f2cb5c1 (127.10.110.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.235806 17424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2eca9fd88c084e51844c3d553f2cb5c1 (127.10.110.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.239028 17562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.239270 17562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:47.239135 17564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.239480 17564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:47.240382 17568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.240546 17568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:37:47.240535 17537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:47.240801 17537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } }
I20260926 16:37:47.249150 17537 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111), 2eca9fd88c084e51844c3d553f2cb5c1 (127.10.110.61:44281)
I20260926 16:37:47.250232 17378 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:47.250309 16827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" is_pre_election: true
I20260926 16:37:47.250608 17378 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:47.250770 17378 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: stopping tablet replica
I20260926 16:37:47.251125 17424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" is_pre_election: true
I20260926 16:37:47.253489 17482 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a1cb8fe9d814ed1a040461f1fced1e3; no voters: 2eca9fd88c084e51844c3d553f2cb5c1, 391f2ba7fa4c47f3bad51f6035a9113e
W20260926 16:37:47.253800 17482 raft_consensus.cc:2670] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260926 16:37:47.257781 17378 master.cc:552] Master@127.10.110.60:42269 shutdown complete.
I20260926 16:37:47.257858 17378 master_runner.cc:305] Clearing existing system tablet
I20260926 16:37:47.258255 17378 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:47.258373 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:47.261206 17378 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 16:37:47.262748 17570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:47.263155 17378 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:47.263404 17378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 16:37:47.263528 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "58a7a5b5f974ea4bb6ca33dd61da2a29"
server_key_iv: "558e37afcdad1f139a6a4c2980c5b8aa"
server_key_version: "encryptionkey@0"
I20260926 16:37:47.268443 17378 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:47.275228 17378 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:37:47.275396 17378 log.cc:1196] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 16:37:47.275898 17378 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Deleting consensus metadata
I20260926 16:37:47.276154 17378 master_runner.cc:315] Copying system tablet from 127.10.110.62:34111
I20260926 16:37:47.278656 17378 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: tablet copy: Beginning tablet copy session from remote peer at address 127.10.110.62:34111
I20260926 16:37:47.281539 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16878
I20260926 16:37:47.296207 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.279656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.10.110.62:34111)
0926 16:37:47.280052 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.280065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.280488 (+ 423us) server_negotiation.cc:407] Connection header received
0926 16:37:47.280672 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.280677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.280756 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.280839 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.283589 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.284755 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.286577 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.287240 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.287400 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.289536 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.289542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.289548 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.289625 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.293294 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.294044 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.294050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.294054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.294203 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.294596 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.294601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.294604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.295020 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.295173 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.295584 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.295808 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260926 16:37:47.296226 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.279582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:41878)
0926 16:37:47.280383 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.280402 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.280545 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.280892 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.280896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.281401 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.283414 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.283433 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.284927 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.284933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.286397 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.286412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.287417 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.287423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.287601 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.288301 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.288323 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.289371 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.293450 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.293455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.293459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.293932 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.294320 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.294324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.294328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.294490 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.295147 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.295151 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.295322 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.295461 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.295763 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":535,"threads_started":1}
I20260926 16:37:47.297071 16837 tablet_copy_service.cc:140] P 391f2ba7fa4c47f3bad51f6035a9113e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7a1cb8fe9d814ed1a040461f1fced1e3 ({username='oryx', principal='oryx/127.10.110.60@KRBTEST.COM'} at 127.0.0.1:41878)
I20260926 16:37:47.297228 16837 tablet_copy_service.cc:161] P 391f2ba7fa4c47f3bad51f6035a9113e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7a1cb8fe9d814ed1a040461f1fced1e3 at {username='oryx', principal='oryx/127.10.110.60@KRBTEST.COM'} at 127.0.0.1:41878: session id = 7a1cb8fe9d814ed1a040461f1fced1e3-00000000000000000000000000000000
I20260926 16:37:47.297307 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:36447
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:34111
--webserver_port=45157
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:37:47.301210 16837 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:37:47.302979 17378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:37:47.308247 17378 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: tablet copy: Starting download of 0 data blocks...
I20260926 16:37:47.308516 17378 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: tablet copy: Starting download of 1 WAL segments...
I20260926 16:37:47.310848 17378 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:37:47.314169 16837 tablet_copy_service.cc:342] P 391f2ba7fa4c47f3bad51f6035a9113e: Request end of tablet copy session 7a1cb8fe9d814ed1a040461f1fced1e3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.110.60@KRBTEST.COM'} at 127.0.0.1:41878
I20260926 16:37:47.314296 16837 tablet_copy_service.cc:433] P 391f2ba7fa4c47f3bad51f6035a9113e: ending tablet copy session 7a1cb8fe9d814ed1a040461f1fced1e3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7a1cb8fe9d814ed1a040461f1fced1e3
I20260926 16:37:47.324172 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:47.326395 17578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:47.329022 17579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:47.343410 17581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:47.343699 17378 server_base.cc:1034] running on GCE node
I20260926 16:37:47.344014 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:47.344606 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:47.345798 17378 hybrid_clock.cc:648] HybridClock initialized: now 1790440667345757 us; error 34 us; skew 500 ppm
I20260926 16:37:47.346006 17378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440667, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:47.347677 17378 init.cc:377] Logged in from keytab as oryx/127.10.110.60@KRBTEST.COM (short username oryx)
I20260926 16:37:47.356734 17378 webserver.cc:460] Webserver started at http://127.10.110.60:40575/ using document root <none> and password file <none>
I20260926 16:37:47.357012 17378 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:47.357100 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:47.359381 17378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 16:37:47.360924 17588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:47.361248 17378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 16:37:47.361352 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
format_stamp: "Formatted at 2026-09-26 16:37:46 on dist-test-slave-whjp"
server_key: "58a7a5b5f974ea4bb6ca33dd61da2a29"
server_key_iv: "558e37afcdad1f139a6a4c2980c5b8aa"
server_key_version: "encryptionkey@0"
I20260926 16:37:47.361513 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:47.370730 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:47.373835 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:47.398365 17378 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.60:42269
I20260926 16:37:47.398419 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.60:42269 every 8 connection(s)
I20260926 16:37:47.399199 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
I20260926 16:37:47.404508 17640 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:37:47.407604 17640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Bootstrap starting.
I20260926 16:37:47.420866 17640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:37:47.421269 17640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: Bootstrap complete.
I20260926 16:37:47.422099 17640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } }
I20260926 16:37:47.422334 17640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7a1cb8fe9d814ed1a040461f1fced1e3, State: Initialized, Role: LEARNER
I20260926 16:37:47.422569 17640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } }
I20260926 16:37:47.423261 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.423561 17640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:47.423739 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:37:47.431023 17654 catalog_manager.cc:1259] Loaded cluster ID: d8939649e04d429499593967064a2623
I20260926 16:37:47.431108 17654 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: loading cluster ID for follower catalog manager: success
I20260926 16:37:47.434429 17654 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: acquiring CA information for follower catalog manager: success
I20260926 16:37:47.436687 17654 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:37:47.625623 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.619720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.0.0.1:37772)
0926 16:37:47.619934 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.619951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.620073 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.620843 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.620848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.620875 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:47.621059 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.621067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.622287 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.622292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.623492 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.623506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.624785 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.624792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.625002 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.625027 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.625143 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.625282 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 16:37:47.625664 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.619914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.10.110.60:42269)
0926 16:37:47.620531 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.620536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.620555 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:37:47.620621 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.620625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.620692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.620800 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:47.621197 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.622147 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.623625 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.624606 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.624718 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.624949 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.625183 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.625315 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
I20260926 16:37:47.626657 17615 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 16:37:47.627213 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391f2ba7fa4c47f3bad51f6035a9113e. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:47.627466 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: This master's current role is: LEARNER
W20260926 16:37:47.654673 17577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:47.654961 17577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:47.655081 17577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:47.662295 17577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:47.662420 17577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:47.662565 17577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:47.672402 17577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:36447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=45157
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:47.673619 17577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:47.674857 17577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:47.687307 17661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:47.687455 17663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:47.687748 17660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:47.688311 17577 server_base.cc:1034] running on GCE node
I20260926 16:37:47.688867 17577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:47.689697 17577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:47.690886 17577 hybrid_clock.cc:648] HybridClock initialized: now 1790440667690841 us; error 34 us; skew 500 ppm
I20260926 16:37:47.691155 17577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440667, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:47.693975 17577 init.cc:377] Logged in from keytab as oryx/127.10.110.1@KRBTEST.COM (short username oryx)
I20260926 16:37:47.696003 17577 webserver.cc:460] Webserver started at http://127.10.110.1:45157/ using document root <none> and password file <none>
I20260926 16:37:47.696662 17577 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:47.696781 17577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:47.701907 17577 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 16:37:47.704506 17670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:47.705829 17577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:47.705973 17577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "72159ff4d4ec48ffaa44b29765a6d720"
format_stamp: "Formatted at 2026-09-26 16:37:44 on dist-test-slave-whjp"
server_key: "8f614ee3dcc6111bd8b227fcdc4056d5"
server_key_iv: "de2633e76d7500849ee3095dda240a5d"
server_key_version: "encryptionkey@0"
I20260926 16:37:47.707265 17577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:47.724942 17577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:47.728636 17577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:47.728861 17577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:47.729707 17577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:47.730940 17577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:47.731019 17577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:47.731094 17577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:47.731146 17577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:47.762642 17577 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:36447
I20260926 16:37:47.762691 17782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:36447 every 8 connection(s)
I20260926 16:37:47.764046 17577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:47.770330 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17577
I20260926 16:37:47.770677 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17012
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440667, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.60@KRBTEST.COM
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440667, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:47.798235 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.767640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:50123 (local address 127.10.110.60:42269)
0926 16:37:47.767893 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.767900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.770223 (+ 2323us) server_negotiation.cc:407] Connection header received
0926 16:37:47.776050 (+ 5827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.776058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.776156 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.776229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.780799 (+ 4570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.781962 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.785295 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.785996 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.786118 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.791109 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.791116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.791121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.791165 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.794705 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.795594 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.795600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.795604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.795735 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.796282 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.796288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.796291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.796621 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.796745 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.797677 (+ 932us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.797905 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.2@KRBTEST.COM
Sep 26 16:37:47 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440667, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
I20260926 16:37:47.800482 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.767986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.10.110.1:50123)
0926 16:37:47.770113 (+ 2127us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.770130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.773098 (+ 2968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.776858 (+ 3760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.776872 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.778800 (+ 1928us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.779552 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.779578 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.782209 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.782215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.785130 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.785142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.786386 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.786392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.786607 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.787564 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.787600 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.790826 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.794870 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.794879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.794896 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.795461 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.795904 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.795909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.795913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.796154 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.796852 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.796861 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.797265 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.798434 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.798992 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2008,"thread_start_us":3891,"threads_started":1}
I20260926 16:37:47.802385 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.768280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48357 (local address 127.10.110.62:34111)
0926 16:37:47.768432 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.768437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.770060 (+ 1623us) server_negotiation.cc:407] Connection header received
0926 16:37:47.777005 (+ 6945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.777012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.777098 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.777164 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.781453 (+ 4289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.782538 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.788853 (+ 6315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.789521 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.789646 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.794237 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.794243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.794248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.794289 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.797532 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.800040 (+ 2508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.800045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.800049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.800207 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.800853 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.800858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.800861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.801203 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.801363 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.801844 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.802080 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:37:47.802526 17783 heartbeater.cc:347] Connected to a master server at 127.10.110.60:42269
I20260926 16:37:47.802842 17783 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:47.803303 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.767730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:48357)
0926 16:37:47.769869 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.769944 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.776661 (+ 6717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.777374 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.777379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.778656 (+ 1277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.780963 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.780977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.782730 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.782735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.786074 (+ 3339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.786089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.789689 (+ 3600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.789695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.789865 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.790461 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.790482 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.794046 (+ 3564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.797656 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.797660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.797664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.799910 (+ 2246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.800348 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.800352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.800356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.800498 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.801422 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.801426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.801567 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.802615 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.802905 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"thread_start_us":1282,"threads_started":1}
I20260926 16:37:47.803759 17783 heartbeater.cc:511] Master 127.10.110.60:42269 requested a full tablet report, sending...
I20260926 16:37:47.804207 17784 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34111
I20260926 16:37:47.804373 17784 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:47.804906 17784 heartbeater.cc:511] Master 127.10.110.62:34111 requested a full tablet report, sending...
I20260926 16:37:47.806023 16817 ts_manager.cc:194] Re-registered known tserver with Master: 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1:36447)
I20260926 16:37:47.806208 17605 ts_manager.cc:194] Registered new tserver with Master: 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1:36447)
I20260926 16:37:47.807098 16817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:48357
I20260926 16:37:47.809087 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.771882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.10.110.2:46117)
0926 16:37:47.787318 (+ 15436us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.787327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.787359 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:37:47.787423 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.787432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.787703 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.787887 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.789721 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.791613 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.793417 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.794214 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.794350 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.800003 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.800017 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.800034 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.800079 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.803291 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.804427 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.804436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.804439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.804620 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.805082 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.805087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.805090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.806981 (+ 1891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.807300 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.808448 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.808709 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15273,"thread_start_us":15129,"threads_started":1}
I20260926 16:37:47.810642 17135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:47.810987 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.770406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.10.110.1:50375)
0926 16:37:47.787962 (+ 17556us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:47.787980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:47.788086 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:47.792904 (+ 4818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:47.792909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:47.793645 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:47.793859 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.793868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.795150 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.795156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.796649 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:47.796664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.797628 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.797634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:47.797810 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.798302 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:47.798323 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:47.804978 (+ 6655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:47.808660 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.808665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.808669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.809027 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.809411 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:47.809415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:47.809418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:47.809559 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:47.810049 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:47.810054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:47.810198 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 16:37:47.810301 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:37:47.810595 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17439,"thread_start_us":16316,"threads_started":1}
I20260926 16:37:47.811017 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:47.770386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:50375 (local address 127.10.110.61:44281)
0926 16:37:47.791641 (+ 21255us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:47.791648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:47.792567 (+ 919us) server_negotiation.cc:407] Connection header received
0926 16:37:47.792635 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:47.792640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:47.792738 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:47.793105 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:47.793995 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.795029 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.796812 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:47.797431 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:47.798083 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:47.805231 (+ 7148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:47.805238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:47.805243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:47.805284 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:47.808533 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.809144 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.809149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.809152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.809288 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:47.809671 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:47.809676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:47.809679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:47.809938 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:47.810079 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:47.810429 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 16:37:47.810627 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21122,"thread_start_us":21479,"threads_started":1}
I20260926 16:37:47.811856 17785 heartbeater.cc:347] Connected to a master server at 127.10.110.61:44281
I20260926 16:37:47.811995 17785 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:47.812409 17785 heartbeater.cc:511] Master 127.10.110.61:44281 requested a full tablet report, sending...
I20260926 16:37:47.814479 17414 ts_manager.cc:194] Registered new tserver with Master: 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1:36447)
I20260926 16:37:47.892823 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17012
I20260926 16:37:47.905349 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:46117
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:34111
--webserver_port=33937
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:37:48.094940 17563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: NON_VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: true } } }
I20260926 16:37:48.095305 17563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:48.096736 17795 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: attempting to promote NON_VOTER 7a1cb8fe9d814ed1a040461f1fced1e3 to VOTER
I20260926 16:37:48.097265 17795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } }
I20260926 16:37:48.098932 17424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Refusing update from remote peer 391f2ba7fa4c47f3bad51f6035a9113e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 16:37:48.099040 17615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 LEARNER]: Refusing update from remote peer 391f2ba7fa4c47f3bad51f6035a9113e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 16:37:48.099524 17563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:37:48.099920 17795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:37:48.102573 17795 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.103133 17424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.105077 17615 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.106599 17795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.106837 17795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:48.107129 17795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.107533 17795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:48.107699 17795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.107916 17795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:48.109582 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.109859 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:37:48.116330 17799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:48.116653 17799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 16:37:48.222808 17794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:48.223075 17794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:48.223210 17794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:48.230490 17794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:48.230590 17794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:48.230710 17794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:48.240154 17794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=33937
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:48.241379 17794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:48.242601 17794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:48.254951 17806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:48.255004 17808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:48.254976 17805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:48.255864 17794 server_base.cc:1034] running on GCE node
I20260926 16:37:48.256448 17794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:48.257309 17794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:48.258497 17794 hybrid_clock.cc:648] HybridClock initialized: now 1790440668258450 us; error 40 us; skew 500 ppm
I20260926 16:37:48.258797 17794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:48.261993 17794 init.cc:377] Logged in from keytab as oryx/127.10.110.2@KRBTEST.COM (short username oryx)
I20260926 16:37:48.263516 17794 webserver.cc:460] Webserver started at http://127.10.110.2:33937/ using document root <none> and password file <none>
I20260926 16:37:48.264243 17794 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:48.264374 17794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:48.269614 17794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:37:48.272349 17815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:48.273592 17794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:37:48.273737 17794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "62f56de115b446d1b5ee5054e0b038bd"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "03eb554c566dc034a054e6a60b6f4cc4"
server_key_iv: "c6b140132906ab591ce16eace9ee93fa"
server_key_version: "encryptionkey@0"
I20260926 16:37:48.274642 17794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:48.290820 17794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:48.294618 17794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:48.294874 17794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:48.295750 17794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:48.297044 17794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:48.297127 17794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:48.297222 17794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:48.297289 17794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:48.325367 17794 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:46117
I20260926 16:37:48.325454 17927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:46117 every 8 connection(s)
I20260926 16:37:48.326742 17794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:48.332652 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17794
I20260926 16:37:48.332959 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17145
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.60@KRBTEST.COM
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.3@KRBTEST.COM
I20260926 16:37:48.357199 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.329512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:46045 (local address 127.10.110.60:42269)
0926 16:37:48.330022 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.330029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.335372 (+ 5343us) server_negotiation.cc:407] Connection header received
0926 16:37:48.335444 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.335451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.335531 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.335622 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.338061 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.340240 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.342770 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.343408 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.343522 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.349228 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.349234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.349238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.349274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.353306 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.354527 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.354533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.354537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.354692 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.355159 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.355164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.355167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.355493 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.355652 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.356611 (+ 959us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.356819 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":232,"threads_started":1}
I20260926 16:37:48.358291 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.330029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.10.110.2:46045)
0926 16:37:48.334610 (+ 4581us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:48.334628 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:48.334751 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:48.336084 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:48.336096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:48.336980 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:48.337837 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.337864 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.340417 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.340423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.342587 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.342602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.344376 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.344382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.345164 (+ 782us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.345709 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:48.345730 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:48.348986 (+ 3256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:48.353752 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.353756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.353760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.354383 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.354799 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.354804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.354808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.355039 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.355723 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:48.355730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:48.356199 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 16:37:48.356504 (+ 305us) client_negotiation.cc:240] Negotiation successful
0926 16:37:48.357024 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4384,"mutex_wait_us":72,"thread_start_us":4162,"threads_started":1}
I20260926 16:37:48.359966 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.330753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.10.110.2:54943)
0926 16:37:48.335164 (+ 4411us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:48.335180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:48.335277 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:48.344547 (+ 9270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:48.344552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:48.345219 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:48.345543 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.345554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.346882 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.346888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.348476 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.348490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.350354 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.350360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.350530 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.351047 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:48.351069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:48.353924 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:48.357470 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.357475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.357479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.357915 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.358334 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.358339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.358342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.358493 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.359049 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:48.359054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:48.359202 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 16:37:48.359328 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 16:37:48.359562 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4302,"thread_start_us":4183,"threads_started":1}
I20260926 16:37:48.360188 17928 heartbeater.cc:347] Connected to a master server at 127.10.110.60:42269
I20260926 16:37:48.360502 17928 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:48.360707 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.329582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:37393 (local address 127.10.110.62:34111)
0926 16:37:48.330349 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.330355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.338672 (+ 8317us) server_negotiation.cc:407] Connection header received
0926 16:37:48.338759 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.338765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.338876 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.338998 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.339890 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.340725 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.342771 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.343400 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.343522 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.349207 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.349213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.349218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.349260 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.352580 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.355239 (+ 2659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.355244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.355248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.355398 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.359100 (+ 3702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.359106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.359110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.359442 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.359575 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.359993 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.360374 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":755,"threads_started":1}
I20260926 16:37:48.360883 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.330634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:54943 (local address 127.10.110.61:44281)
0926 16:37:48.344194 (+ 13560us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.344200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.344218 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:37:48.344273 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.344278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.344375 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.344987 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.345728 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.346718 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.348670 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.350149 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.350795 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.354148 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.354155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.354160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.354214 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.357315 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.358072 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.358078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.358081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.358210 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.358632 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.358637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.358640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.358915 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.360211 (+ 1296us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.360341 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.360575 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13437,"thread_start_us":13342,"threads_started":1}
I20260926 16:37:48.361148 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.330029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:37393)
0926 16:37:48.330979 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:48.331039 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:48.333801 (+ 2762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:48.339028 (+ 5227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:48.339034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:48.339436 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:48.339713 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.339724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.340868 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.340874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.342620 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.342636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.343618 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.343625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.343978 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.344699 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:48.344734 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:48.348987 (+ 4253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:48.353526 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.353537 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.353561 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.354383 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.356153 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.356158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.356161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.356309 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.359553 (+ 3244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:48.359558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:48.359722 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 16:37:48.359885 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 16:37:48.360665 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":38,"thread_start_us":489,"threads_started":1}
I20260926 16:37:48.361392 17928 heartbeater.cc:511] Master 127.10.110.60:42269 requested a full tablet report, sending...
I20260926 16:37:48.361999 17929 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34111
I20260926 16:37:48.362126 17929 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:48.362303 17930 heartbeater.cc:347] Connected to a master server at 127.10.110.61:44281
I20260926 16:37:48.362426 17930 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:48.362638 17929 heartbeater.cc:511] Master 127.10.110.62:34111 requested a full tablet report, sending...
I20260926 16:37:48.362831 17930 heartbeater.cc:511] Master 127.10.110.61:44281 requested a full tablet report, sending...
I20260926 16:37:48.363718 16817 ts_manager.cc:194] Re-registered known tserver with Master: 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:48.364428 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.333853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.10.110.3:44193)
0926 16:37:48.339582 (+ 5729us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.339592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.341288 (+ 1696us) server_negotiation.cc:407] Connection header received
0926 16:37:48.341356 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.341364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.341614 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.341751 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.344760 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.346409 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.348021 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.348863 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.348987 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.355331 (+ 6344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.355361 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.355380 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.355420 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.358943 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.361422 (+ 2479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.361430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.361433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.361578 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.362011 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.362016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.362020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.362500 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.362787 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.363789 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.364083 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5541,"thread_start_us":5385,"threads_started":1}
I20260926 16:37:48.364711 17605 ts_manager.cc:194] Registered new tserver with Master: 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:48.364885 16817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:37393
I20260926 16:37:48.365329 17414 ts_manager.cc:194] Registered new tserver with Master: 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:48.366380 17268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:48.451925 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17145
I20260926 16:37:48.464414 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:44193
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:34111
--webserver_port=37599
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:48.781912 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:48.782204 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:48.782344 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:48.789958 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:48.790088 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:48.790256 17941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:48.800130 17941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:44193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=37599
--tserver_master_addrs=127.10.110.62:34111,127.10.110.61:44281,127.10.110.60:42269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:48.801468 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:48.802897 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:37:48.809754 17784 heartbeater.cc:503] Master 127.10.110.62:34111 was elected leader, sending a full tablet report...
W20260926 16:37:48.816085 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:48.816121 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:48.816274 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:48.816900 17941 server_base.cc:1034] running on GCE node
I20260926 16:37:48.817457 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:48.818336 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:48.819530 17941 hybrid_clock.cc:648] HybridClock initialized: now 1790440668819471 us; error 51 us; skew 500 ppm
I20260926 16:37:48.819816 17941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:48.822765 17941 init.cc:377] Logged in from keytab as oryx/127.10.110.3@KRBTEST.COM (short username oryx)
I20260926 16:37:48.824188 17941 webserver.cc:460] Webserver started at http://127.10.110.3:37599/ using document root <none> and password file <none>
I20260926 16:37:48.824856 17941 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:48.824961 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:48.830148 17941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:37:48.832682 17956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:48.833945 17941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:37:48.834077 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "c63c9ec9560440b1bb66ea08b359a00e"
format_stamp: "Formatted at 2026-09-26 16:37:45 on dist-test-slave-whjp"
server_key: "c606ace2444509f60fc1de1d2402f5f7"
server_key_iv: "ca58ff11470deb1e2a037e8249d45a54"
server_key_version: "encryptionkey@0"
I20260926 16:37:48.834981 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:48.862943 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:48.866840 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:48.867096 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:48.868009 17941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:37:48.869315 17941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:48.869406 17941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:48.869503 17941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:48.869568 17941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:48.897995 17941 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:44193
I20260926 16:37:48.898048 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:44193 every 8 connection(s)
I20260926 16:37:48.899324 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:48.903445 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 17941
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.61@KRBTEST.COM
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
Sep 26 16:37:48 dist-test-slave-whjp krb5kdc[16772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.60@KRBTEST.COM
I20260926 16:37:48.938714 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.902629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.10.110.3:38977)
0926 16:37:48.905028 (+ 2399us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:48.905087 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:48.909838 (+ 4751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:48.914978 (+ 5140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:48.914982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:48.915996 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:48.917040 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.917066 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.921248 (+ 4182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.921251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.922469 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.922480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.924431 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.924438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.924603 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.925170 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:48.925191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:48.928189 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:48.932602 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.932615 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.932639 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.933305 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.934495 (+ 1190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.934501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.934505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.934777 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.935577 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:48.935589 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:48.936139 (+ 550us) client_negotiation.cc:769] Sending connection context
0926 16:37:48.936474 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 16:37:48.937090 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"mutex_wait_us":112,"spinlock_wait_cycles":19328,"thread_start_us":2198,"threads_started":1}
I20260926 16:37:48.940529 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.902112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:38977 (local address 127.10.110.60:42269)
0926 16:37:48.914669 (+ 12557us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.914675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.914692 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:37:48.914745 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.914749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.914816 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.914932 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.917261 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.918473 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.922606 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.923496 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.923618 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.928430 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.928438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.928444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.928489 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.932370 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.933492 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.933499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.933504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.933670 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.934914 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.934921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.934925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.935313 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.935493 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.939283 (+ 3790us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.940100 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12449,"thread_start_us":12349,"threads_started":1}
I20260926 16:37:48.942996 18069 heartbeater.cc:347] Connected to a master server at 127.10.110.60:42269
I20260926 16:37:48.943393 18069 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:48.944335 18069 heartbeater.cc:511] Master 127.10.110.60:42269 requested a full tablet report, sending...
I20260926 16:37:48.945178 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.903034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:51355)
0926 16:37:48.914141 (+ 11107us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:48.914160 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:48.914281 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:48.914919 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:48.914932 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:48.916002 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:48.917039 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.917066 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.919966 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.919988 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.921130 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.921142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.922674 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.922683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.922863 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.923432 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:48.923455 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:48.927791 (+ 4336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:48.933813 (+ 6022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.933819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.933823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.934368 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.943034 (+ 8666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.943040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.943044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.943267 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.944021 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:48.944027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:48.944207 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 16:37:48.944406 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 16:37:48.944725 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10987,"mutex_wait_us":167,"thread_start_us":9706,"threads_started":1}
I20260926 16:37:48.945897 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.902738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:51355 (local address 127.10.110.62:34111)
0926 16:37:48.912210 (+ 9472us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.912217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.914265 (+ 2048us) server_negotiation.cc:407] Connection header received
0926 16:37:48.914402 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.914409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.914510 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.914634 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.918666 (+ 4032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.919823 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.921401 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.921969 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.922094 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.928076 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.928083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.928088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.928135 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.933448 (+ 5313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.941857 (+ 8409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.941876 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.941881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.942068 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.943490 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.943496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.943500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.943905 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.944066 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.945154 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.945538 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9332,"thread_start_us":9239,"threads_started":1}
I20260926 16:37:48.946456 17605 ts_manager.cc:194] Registered new tserver with Master: c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193)
I20260926 16:37:48.947170 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.902396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:50313 (local address 127.10.110.61:44281)
0926 16:37:48.912569 (+ 10173us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.912576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.913300 (+ 724us) server_negotiation.cc:407] Connection header received
0926 16:37:48.913372 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.913378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.913479 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.913576 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.917245 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.917974 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.919583 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.920336 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.920462 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.935951 (+ 15489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.935959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.935965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.936012 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.940640 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.942835 (+ 2195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.942841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.942845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.943008 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.944152 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.944157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.944160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.944498 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.944645 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.946524 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.946818 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10047,"thread_start_us":9938,"threads_started":1}
I20260926 16:37:48.948477 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.902818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.10.110.3:50313)
0926 16:37:48.913028 (+ 10210us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:48.913048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:48.913185 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:48.915094 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:48.915098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:48.916002 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:48.917039 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.917066 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.918123 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.918128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.919446 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:48.919458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.920585 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.920593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:48.920921 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.922376 (+ 1455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:48.922408 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:48.934882 (+ 12474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:48.940922 (+ 6040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.940928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.940933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.941508 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.943174 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:48.943179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:48.943182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:48.943360 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:48.944721 (+ 1361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:48.944726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:48.944875 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 16:37:48.947696 (+ 2821us) client_negotiation.cc:240] Negotiation successful
0926 16:37:48.948037 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10076,"mutex_wait_us":85,"spinlock_wait_cycles":30336,"thread_start_us":11007,"threads_started":1}
I20260926 16:37:48.949441 18070 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34111
I20260926 16:37:48.949790 18070 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:48.950296 18070 heartbeater.cc:511] Master 127.10.110.62:34111 requested a full tablet report, sending...
I20260926 16:37:48.951337 16817 ts_manager.cc:194] Re-registered known tserver with Master: c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193)
I20260926 16:37:48.952546 16817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:51355
I20260926 16:37:48.954095 18071 heartbeater.cc:347] Connected to a master server at 127.10.110.61:44281
I20260926 16:37:48.954217 18071 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:48.955575 18071 heartbeater.cc:511] Master 127.10.110.61:44281 requested a full tablet report, sending...
I20260926 16:37:48.956779 17414 ts_manager.cc:194] Registered new tserver with Master: c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193)
I20260926 16:37:48.973718 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.924103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.10.110.62:34111)
0926 16:37:48.949649 (+ 25546us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.949655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.955357 (+ 5702us) server_negotiation.cc:407] Connection header received
0926 16:37:48.955415 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.955420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.955509 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.955586 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.959542 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.960593 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.962368 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.962940 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.963052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.966227 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.966234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.966239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.966280 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.969267 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.969856 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.969862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.969866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.970020 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.972063 (+ 2043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.972069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.972072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.972432 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.972656 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.972981 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.973358 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25442,"thread_start_us":28793,"threads_started":1}
I20260926 16:37:48.974378 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.937798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.10.110.61:44281)
0926 16:37:48.947657 (+ 9859us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.947663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.959896 (+ 12233us) server_negotiation.cc:407] Connection header received
0926 16:37:48.960051 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.960056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.960153 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.960266 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.962171 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.962974 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.964912 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.965301 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.965415 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.968382 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.968389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.968394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.968433 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.971508 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.972380 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.972385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.972389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.972530 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.972899 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.972903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.972906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.973241 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.973430 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.973857 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.974072 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9749,"thread_start_us":19300,"threads_started":1}
I20260926 16:37:48.974646 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:48.923901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.10.110.60:42269)
0926 16:37:48.941200 (+ 17299us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:48.941207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:48.958902 (+ 17695us) server_negotiation.cc:407] Connection header received
0926 16:37:48.959088 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:48.959095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:48.959200 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:48.959294 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:48.961663 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.962682 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.964614 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:48.965223 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:48.965364 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:48.968685 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:48.968692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:48.968697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:48.968739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:48.971729 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.972475 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.972481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.972484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.972630 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:48.973016 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:48.973021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:48.973024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:48.973339 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:48.973574 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:48.974115 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 16:37:48.974316 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17183,"thread_start_us":35325,"threads_started":1}
I20260926 16:37:48.978204 16817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:48.985245 16817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:49.016428 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.004388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.10.110.1:36447)
0926 16:37:49.009373 (+ 4985us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.009380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.009402 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:37:49.009481 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.009486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.009667 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.009912 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:49.010378 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.011783 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.014534 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.015424 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.015549 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.015748 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.015991 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.016156 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4844,"thread_start_us":4737,"threads_started":1}
I20260926 16:37:49.016494 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.004176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36447 (local address 127.0.0.1:48884)
0926 16:37:49.008398 (+ 4222us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:49.008417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:49.008534 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:49.009866 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:49.009870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:49.009909 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:49.010191 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.010201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.012043 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.012047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.014410 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.014424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.015627 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.015633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.015789 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.015822 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:49.015932 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:37:49.016066 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4060,"mutex_wait_us":40,"thread_start_us":3867,"threads_started":1}
I20260926 16:37:49.016980 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.003714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.10.110.2:46117)
0926 16:37:49.004241 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.004252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.004282 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:37:49.004392 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.004401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.004701 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.004977 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:49.009778 (+ 4801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.011219 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.014435 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.015343 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.015457 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.015651 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.016555 (+ 904us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.016718 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":605,"threads_started":1}
I20260926 16:37:49.016433 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.003237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46117 (local address 127.0.0.1:45354)
0926 16:37:49.003724 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:49.003740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:49.003878 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:49.009353 (+ 5475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:49.009357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:49.009381 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:49.009587 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.009594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.011379 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.011384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.014279 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.014295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.015525 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.015531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.015695 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.015727 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:37:49.015822 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:37:49.015990 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":57,"thread_start_us":608,"threads_started":1}
I20260926 16:37:49.020160 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.002574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44193 (local address 127.0.0.1:50458)
0926 16:37:49.003219 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:49.003241 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:49.003407 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:49.011489 (+ 8082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:49.011493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:49.011523 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:49.012528 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.012540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.015867 (+ 3327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.015872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.017706 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.017718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.019417 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.019423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.019586 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.019619 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:49.019712 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:37:49.019861 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":331,"threads_started":1}
I20260926 16:37:49.020247 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.002776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.10.110.3:44193)
0926 16:37:49.009125 (+ 6349us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.009134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.009169 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:37:49.009266 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.009276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.009553 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.011441 (+ 1888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:49.012838 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.015734 (+ 2896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.017864 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.019225 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.019350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.019541 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.019758 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.019950 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6193,"thread_start_us":6044,"threads_started":1}
I20260926 16:37:49.020685 17718 tablet_service.cc:1467] Processing CreateTablet for tablet 37e4a546c62f4fcca59222b7cac6ea63 (DEFAULT_TABLE table=test-table [id=dc6ffbebd73d467a9786832f17216147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:49.021749 17863 tablet_service.cc:1467] Processing CreateTablet for tablet 37e4a546c62f4fcca59222b7cac6ea63 (DEFAULT_TABLE table=test-table [id=dc6ffbebd73d467a9786832f17216147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:49.022830 17863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e4a546c62f4fcca59222b7cac6ea63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:49.023123 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 37e4a546c62f4fcca59222b7cac6ea63 (DEFAULT_TABLE table=test-table [id=dc6ffbebd73d467a9786832f17216147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:49.024268 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e4a546c62f4fcca59222b7cac6ea63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:49.025637 17718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e4a546c62f4fcca59222b7cac6ea63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:49.033939 18103 tablet_bootstrap.cc:492] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Bootstrap starting.
I20260926 16:37:49.037050 18103 tablet_bootstrap.cc:654] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:49.038192 18104 tablet_bootstrap.cc:492] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Bootstrap starting.
I20260926 16:37:49.038167 18103 log.cc:824] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:49.041914 18104 tablet_bootstrap.cc:654] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:49.043272 18104 log.cc:824] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:49.044282 18106 tablet_bootstrap.cc:492] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Bootstrap starting.
I20260926 16:37:49.046950 18103 tablet_bootstrap.cc:492] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: No bootstrap required, opened a new log
I20260926 16:37:49.047232 18103 ts_tablet_manager.cc:1397] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260926 16:37:49.047292 18106 tablet_bootstrap.cc:654] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:49.047451 18104 tablet_bootstrap.cc:492] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: No bootstrap required, opened a new log
I20260926 16:37:49.047695 18104 ts_tablet_manager.cc:1397] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 16:37:49.048298 18106 log.cc:824] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:49.050922 18106 tablet_bootstrap.cc:492] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: No bootstrap required, opened a new log
I20260926 16:37:49.051167 18106 ts_tablet_manager.cc:1397] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 16:37:49.057775 18104 raft_consensus.cc:348] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.057934 18106 raft_consensus.cc:348] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.058219 18104 raft_consensus.cc:374] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:49.058231 18106 raft_consensus.cc:374] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:49.058313 18106 raft_consensus.cc:729] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63c9ec9560440b1bb66ea08b359a00e, State: Initialized, Role: FOLLOWER
I20260926 16:37:49.058348 18104 raft_consensus.cc:729] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72159ff4d4ec48ffaa44b29765a6d720, State: Initialized, Role: FOLLOWER
I20260926 16:37:49.058320 18103 raft_consensus.cc:348] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.058715 18103 raft_consensus.cc:374] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:49.058815 18103 raft_consensus.cc:729] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f56de115b446d1b5ee5054e0b038bd, State: Initialized, Role: FOLLOWER
I20260926 16:37:49.058882 18106 consensus_queue.cc:260] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.058993 18104 consensus_queue.cc:260] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.059324 18103 consensus_queue.cc:260] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.060585 18106 ts_tablet_manager.cc:1428] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260926 16:37:49.060875 18070 heartbeater.cc:503] Master 127.10.110.62:34111 was elected leader, sending a full tablet report...
I20260926 16:37:49.063771 17929 heartbeater.cc:503] Master 127.10.110.62:34111 was elected leader, sending a full tablet report...
I20260926 16:37:49.065297 18103 ts_tablet_manager.cc:1428] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260926 16:37:49.070444 18104 ts_tablet_manager.cc:1428] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Time spent starting tablet: real 0.023s user 0.017s sys 0.000s
W20260926 16:37:49.080137 17931 tablet.cc:2369] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:49.139333 18110 raft_consensus.cc:482] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:49.139741 18110 raft_consensus.cc:504] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.141408 18110 leader_election.cc:288] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193), 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:49.149145 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.141697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:35859 (local address 127.10.110.3:44193)
0926 16:37:49.141909 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.141915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.142256 (+ 341us) server_negotiation.cc:407] Connection header received
0926 16:37:49.142477 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.142484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.142562 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.142651 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:49.143234 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.144169 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.146430 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.148093 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.148250 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.148326 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.148795 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.148917 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:37:49.149313 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.141606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44193 (local address 127.10.110.1:35859)
0926 16:37:49.142139 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:49.142155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:49.142337 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:49.142712 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:49.142718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:49.142754 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:49.143101 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.143110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.144292 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.144298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.146251 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.146266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.148407 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.148416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.148611 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.148648 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:37:49.148745 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:37:49.148870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":1,"thread_start_us":284,"threads_started":1}
I20260926 16:37:49.149379 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.142450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:33903 (local address 127.10.110.2:46117)
0926 16:37:49.142641 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.142646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.143090 (+ 444us) server_negotiation.cc:407] Connection header received
0926 16:37:49.143288 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.143292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.143346 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.143409 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:49.143922 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.144884 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.147055 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.148349 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.148464 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.148525 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.149015 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.149137 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:37:49.149483 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.142267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46117 (local address 127.10.110.1:33903)
0926 16:37:49.142999 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:49.143012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:49.143130 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:49.143445 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:49.143449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:49.143472 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:49.143750 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.143761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.145038 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.145042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.146905 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:49.146920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.148699 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.148707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:49.148867 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.148891 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:37:49.148964 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 16:37:49.149087 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":1,"thread_start_us":484,"threads_started":1}
I20260926 16:37:49.150534 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e4a546c62f4fcca59222b7cac6ea63" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63c9ec9560440b1bb66ea08b359a00e" is_pre_election: true
I20260926 16:37:49.150534 17883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e4a546c62f4fcca59222b7cac6ea63" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f56de115b446d1b5ee5054e0b038bd" is_pre_election: true
I20260926 16:37:49.151079 17883 raft_consensus.cc:2454] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 0.
I20260926 16:37:49.151077 18024 raft_consensus.cc:2454] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 0.
I20260926 16:37:49.151706 17674 leader_election.cc:302] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f56de115b446d1b5ee5054e0b038bd, 72159ff4d4ec48ffaa44b29765a6d720; no voters:
I20260926 16:37:49.152127 18110 raft_consensus.cc:2781] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:49.152259 18110 raft_consensus.cc:482] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:49.152436 18110 raft_consensus.cc:3037] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:49.154546 18110 raft_consensus.cc:504] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:49.155076 18110 leader_election.cc:288] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 election: Requested vote from peers c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193), 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:49.155536 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e4a546c62f4fcca59222b7cac6ea63" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63c9ec9560440b1bb66ea08b359a00e"
I20260926 16:37:49.155591 17883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e4a546c62f4fcca59222b7cac6ea63" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f56de115b446d1b5ee5054e0b038bd"
I20260926 16:37:49.155746 18024 raft_consensus.cc:3037] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:49.155768 17883 raft_consensus.cc:3037] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:49.158814 18024 raft_consensus.cc:2454] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 1.
I20260926 16:37:49.158814 17883 raft_consensus.cc:2454] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 1.
I20260926 16:37:49.159152 17674 leader_election.cc:302] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f56de115b446d1b5ee5054e0b038bd, 72159ff4d4ec48ffaa44b29765a6d720; no voters:
I20260926 16:37:49.159399 18110 raft_consensus.cc:2781] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:49.159785 18110 raft_consensus.cc:686] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 LEADER]: Becoming Leader. State: Replica: 72159ff4d4ec48ffaa44b29765a6d720, State: Running, Role: LEADER
I20260926 16:37:49.160307 18110 consensus_queue.cc:237] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
W20260926 16:37:49.160625 18072 tablet.cc:2369] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:49.165602 16815 catalog_manager.cc:5495] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "72159ff4d4ec48ffaa44b29765a6d720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260926 16:37:49.206053 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.201380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.10.110.1:36447)
0926 16:37:49.201536 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.201541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.201557 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:37:49.201640 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.201643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.201716 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.201852 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:49.202120 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.202780 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.204135 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.204601 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.204688 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.205093 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:49.205563 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:49.205665 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.205729 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.205793 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:37:49.214311 17883 raft_consensus.cc:1261] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Refusing update from remote peer 72159ff4d4ec48ffaa44b29765a6d720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:49.214313 18024 raft_consensus.cc:1261] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Refusing update from remote peer 72159ff4d4ec48ffaa44b29765a6d720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:49.215164 18114 consensus_queue.cc:1035] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:49.215523 18110 consensus_queue.cc:1035] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:49.243187 16815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:49.243433 16815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:49.249804 16815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 391f2ba7fa4c47f3bad51f6035a9113e: Sending DeleteTablet for 3 replicas of tablet 37e4a546c62f4fcca59222b7cac6ea63
I20260926 16:37:49.250881 18004 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e4a546c62f4fcca59222b7cac6ea63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:49 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:50458
I20260926 16:37:49.251009 17863 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e4a546c62f4fcca59222b7cac6ea63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:49 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:45354
I20260926 16:37:49.251914 17718 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e4a546c62f4fcca59222b7cac6ea63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:49 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:48884
I20260926 16:37:49.252136 18127 tablet_replica.cc:330] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: stopping tablet replica
I20260926 16:37:49.252544 18127 raft_consensus.cc:2229] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:49.253065 18127 raft_consensus.cc:2258] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:49.259687 18127 ts_tablet_manager.cc:1905] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:49.263548 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 16785
I20260926 16:37:49.263641 18129 tablet_replica.cc:330] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: stopping tablet replica
I20260926 16:37:49.263770 18128 tablet_replica.cc:330] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: stopping tablet replica
I20260926 16:37:49.264067 18129 raft_consensus.cc:2229] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:49.264276 18128 raft_consensus.cc:2229] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:49.264810 18129 raft_consensus.cc:2258] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:49.264829 18128 raft_consensus.cc:2258] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:49.265769 16847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:49.267019 18129 ts_tablet_manager.cc:1905] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:49.270542 18127 ts_tablet_manager.cc:1918] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:49.270704 18127 log.cc:1196] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/37e4a546c62f4fcca59222b7cac6ea63
I20260926 16:37:49.271252 18127 ts_tablet_manager.cc:1939] T 37e4a546c62f4fcca59222b7cac6ea63 P c63c9ec9560440b1bb66ea08b359a00e: Deleting consensus metadata
I20260926 16:37:49.274432 18128 ts_tablet_manager.cc:1905] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:49.281455 18128 ts_tablet_manager.cc:1918] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:49.281627 18128 log.cc:1196] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/37e4a546c62f4fcca59222b7cac6ea63
I20260926 16:37:49.282150 18128 ts_tablet_manager.cc:1939] T 37e4a546c62f4fcca59222b7cac6ea63 P 62f56de115b446d1b5ee5054e0b038bd: Deleting consensus metadata
I20260926 16:37:49.286377 18129 ts_tablet_manager.cc:1918] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:49.287253 18129 log.cc:1196] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/37e4a546c62f4fcca59222b7cac6ea63
I20260926 16:37:49.287328 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.254061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.10.110.61:44281)
0926 16:37:49.254240 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.254247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.262781 (+ 8534us) server_negotiation.cc:407] Connection header received
0926 16:37:49.262981 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.262987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.263065 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.263164 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:49.273413 (+ 10249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.274489 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.276424 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.277034 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.277131 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.280750 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:49.280757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:49.280762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:49.280815 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:49.284101 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.284961 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.284967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.284971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.285112 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.285599 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.285605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.285608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.285962 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:49.286183 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.286730 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.286950 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:37:49.288215 18129 ts_tablet_manager.cc:1939] T 37e4a546c62f4fcca59222b7cac6ea63 P 72159ff4d4ec48ffaa44b29765a6d720: Deleting consensus metadata
I20260926 16:37:49.302069 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.262442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.10.110.60:42269)
0926 16:37:49.262613 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.262619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.288717 (+ 26098us) server_negotiation.cc:407] Connection header received
0926 16:37:49.288994 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.289001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.289085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.289158 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:49.291412 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.292197 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.293370 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.293749 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.293857 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.296170 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:49.296175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:49.296178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:49.296219 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:49.299054 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.299605 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.299610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.299614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.299729 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.300102 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.300107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.300110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.300379 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:49.301492 (+ 1113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.301638 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.301819 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:37:49.468071 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 16785
W20260926 16:37:49.492990 17674 connection.cc:535] server connection from 127.0.0.1:48884 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 16:37:49.492985 17816 connection.cc:535] server connection from 127.0.0.1:45354 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:37:49.520265 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.497837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.10.110.61:44281)
0926 16:37:49.498018 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.498024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.505080 (+ 7056us) server_negotiation.cc:407] Connection header received
0926 16:37:49.505546 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.505554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.505638 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.505708 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:49.507817 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.508918 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.510996 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.511629 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.511722 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.515222 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:49.515229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:49.515234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:49.515271 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:49.517697 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.518381 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.518387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.518390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.518539 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.518910 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.518916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.518919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.519182 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:49.519398 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.519708 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.519946 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:49.523619 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:49.497485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.10.110.60:42269)
0926 16:37:49.497664 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:49.497670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:49.509119 (+ 11449us) server_negotiation.cc:407] Connection header received
0926 16:37:49.509380 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:49.509387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:49.509470 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:49.509540 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:49.512145 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.513363 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.514633 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:49.515027 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:49.515130 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:49.518024 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:49.518030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:49.518035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:49.518072 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:49.521257 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.521976 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.521980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.521983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.522108 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:49.522435 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:49.522439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:49.522441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:49.522683 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:49.522881 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:49.523144 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 16:37:49.523295 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:50.234061 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.233413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:40121)
0926 16:37:50.233797 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.233906 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":191,"threads_started":1}
W20260926 16:37:50.234520 17929 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:34111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.236325 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.235378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:57017)
0926 16:37:50.236013 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.236138 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":672,"threads_started":1}
W20260926 16:37:50.236701 17784 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:34111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.239604 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.238866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:58389)
0926 16:37:50.239310 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.239430 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":209,"threads_started":1}
W20260926 16:37:50.239991 18070 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:34111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.750138 18142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 391f2ba7fa4c47f3bad51f6035a9113e)
I20260926 16:37:50.750326 18142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } }
I20260926 16:37:50.750706 18143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 391f2ba7fa4c47f3bad51f6035a9113e)
I20260926 16:37:50.750948 18143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } }
I20260926 16:37:50.751199 18142 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111), 2eca9fd88c084e51844c3d553f2cb5c1 (127.10.110.61:44281)
I20260926 16:37:50.761055 18143 leader_election.cc:288] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111), 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60:42269)
I20260926 16:37:50.761365 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.752698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42354)
0926 16:37:50.760804 (+ 8106us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.761086 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7941,"threads_started":1}
I20260926 16:37:50.768167 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.751410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42340)
0926 16:37:50.767578 (+ 16168us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.767859 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16020,"threads_started":1}
W20260926 16:37:50.771622 17592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
W20260926 16:37:50.772293 17401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.772821 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.760046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.10.110.60:42269)
0926 16:37:50.760567 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.760573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.762871 (+ 2298us) server_negotiation.cc:407] Connection header received
0926 16:37:50.763512 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.763518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.763594 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.763697 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.764199 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.765515 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.769745 (+ 4230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.771116 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.771235 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.771349 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.772302 (+ 953us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.772440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":232,"threads_started":1}
I20260926 16:37:50.773505 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.759473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:42269 (local address 127.0.0.1:37810)
0926 16:37:50.762768 (+ 3295us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.762786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.762940 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.763743 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.763748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.763774 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.764067 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.764079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.765632 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.765638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.769499 (+ 3861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.769519 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.771646 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.771656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.771958 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.771993 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.772088 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.773088 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3192,"mutex_wait_us":752,"thread_start_us":1015,"threads_started":1}
I20260926 16:37:50.774470 17615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" is_pre_election: true
I20260926 16:37:50.774866 17615 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eca9fd88c084e51844c3d553f2cb5c1 in term 1.
I20260926 16:37:50.775408 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.775030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42364)
0926 16:37:50.775181 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.775263 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 16:37:50.775528 17398 leader_election.cc:302] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eca9fd88c084e51844c3d553f2cb5c1, 7a1cb8fe9d814ed1a040461f1fced1e3; no voters:
W20260926 16:37:50.775848 17401 leader_election.cc:334] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111): Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.775945 18143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 16:37:50.776154 18143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 391f2ba7fa4c47f3bad51f6035a9113e)
I20260926 16:37:50.776315 18143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:37:50.776322 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.775930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42372)
0926 16:37:50.776080 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.776159 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 16:37:50.776556 17592 leader_election.cc:334] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111): Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.778728 18143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } }
I20260926 16:37:50.779192 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.751613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.10.110.61:44281)
0926 16:37:50.771774 (+ 20161us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.771780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.771802 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:37:50.771889 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.771893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.771966 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.772069 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.772978 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.774406 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.777178 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.778214 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.778336 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.778448 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.778700 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.778831 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20006,"thread_start_us":20510,"threads_started":1}
I20260926 16:37:50.779198 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.751460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:44281 (local address 127.0.0.1:43318)
0926 16:37:50.768871 (+ 17411us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.768889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.769002 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.772373 (+ 3371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.772379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.772410 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.772787 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.772798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.774603 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.774609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.777016 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.777029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.778389 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.778395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.778554 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.778578 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.778656 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.778875 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17304,"thread_start_us":16884,"threads_started":1}
I20260926 16:37:50.779585 18143 leader_election.cc:288] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [CANDIDATE]: Term 2 election: Requested vote from peers 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111), 7a1cb8fe9d814ed1a040461f1fced1e3 (127.10.110.60:42269)
I20260926 16:37:50.779847 17615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3"
I20260926 16:37:50.780062 17615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:37:50.779745 17424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" is_pre_election: true
I20260926 16:37:50.780408 17424 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a1cb8fe9d814ed1a040461f1fced1e3 in current term 2: Already voted for candidate 2eca9fd88c084e51844c3d553f2cb5c1 in this term.
I20260926 16:37:50.780067 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.779605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42380)
0926 16:37:50.779853 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.779937 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:37:50.780922 17590 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a1cb8fe9d814ed1a040461f1fced1e3; no voters: 2eca9fd88c084e51844c3d553f2cb5c1, 391f2ba7fa4c47f3bad51f6035a9113e
I20260926 16:37:50.781572 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.781194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42384)
0926 16:37:50.781359 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.781441 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 16:37:50.781741 17401 leader_election.cc:334] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111): Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
I20260926 16:37:50.782646 17615 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eca9fd88c084e51844c3d553f2cb5c1 in term 2.
I20260926 16:37:50.782823 18142 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:37:50.783033 17398 leader_election.cc:302] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2eca9fd88c084e51844c3d553f2cb5c1, 7a1cb8fe9d814ed1a040461f1fced1e3; no voters: 391f2ba7fa4c47f3bad51f6035a9113e
I20260926 16:37:50.783330 18143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:37:50.783735 18143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [term 2 LEADER]: Becoming Leader. State: Replica: 2eca9fd88c084e51844c3d553f2cb5c1, State: Running, Role: LEADER
I20260926 16:37:50.784106 18143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } }
I20260926 16:37:50.785550 18152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eca9fd88c084e51844c3d553f2cb5c1. Latest consensus state: current_term: 2 leader_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:50.785861 18152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:50.786500 18154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:50.790587 18154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:50.791414 18154 catalog_manager.cc:1259] Loaded cluster ID: d8939649e04d429499593967064a2623
I20260926 16:37:50.791488 18154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:50.792600 18154 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:50.793349 18154 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Loaded TSK: 0
I20260926 16:37:50.794142 18154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:37:50.848014 17413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:50.854902 17413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:50.859022 17615 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [term 2 FOLLOWER]: Refusing update from remote peer 2eca9fd88c084e51844c3d553f2cb5c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 16:37:50.859732 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.858683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42406)
0926 16:37:50.859454 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.859561 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654}
I20260926 16:37:50.859714 18152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 16:37:50.860737 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.860395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42418)
0926 16:37:50.860537 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.860630 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 16:37:50.861002 17401 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 -> Peer 391f2ba7fa4c47f3bad51f6035a9113e (127.10.110.62:34111): Couldn't send request to peer 391f2ba7fa4c47f3bad51f6035a9113e. Status: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 16:37:50.862452 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eca9fd88c084e51844c3d553f2cb5c1. Latest consensus state: current_term: 2 leader_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:50.862715 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:37:50.863937 18152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:50.864192 18152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:50.864711 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "391f2ba7fa4c47f3bad51f6035a9113e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34111 } } peers { permanent_uuid: "2eca9fd88c084e51844c3d553f2cb5c1" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 44281 } attrs { promote: false } } peers { permanent_uuid: "7a1cb8fe9d814ed1a040461f1fced1e3" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 42269 } attrs { promote: false } } }
I20260926 16:37:50.864962 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1cb8fe9d814ed1a040461f1fced1e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:37:50.880596 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.872124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44193 (local address 127.0.0.1:50472)
0926 16:37:50.873221 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.873243 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.873359 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.873786 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.873791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.873817 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.874212 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.874223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.875787 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.875793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.878046 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.878062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.879795 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.879802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.880014 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.880053 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.880121 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.880253 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974}
I20260926 16:37:50.881690 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.872398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.10.110.3:44193)
0926 16:37:50.872878 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.872882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.873472 (+ 590us) server_negotiation.cc:407] Connection header received
0926 16:37:50.873551 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.873556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.873640 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.874380 (+ 740us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.874564 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.875641 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.878254 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.879557 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.881072 (+ 1515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.881185 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.881234 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.881371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
I20260926 16:37:50.892235 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.873065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46117 (local address 127.0.0.1:45364)
0926 16:37:50.880801 (+ 7736us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.880818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.880921 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.884455 (+ 3534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.884460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.884490 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.884661 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.884671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.886471 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.886477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.889931 (+ 3454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.889941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.891459 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.891465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.891663 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.891698 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.891789 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.891965 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7647}
I20260926 16:37:50.892726 18004 tablet_service.cc:1467] Processing CreateTablet for tablet 8a67c366512145bfb18da40e3f8e522f (DEFAULT_TABLE table=test-table [id=a5eb61c8cfdf4955a396b342fa7eb130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:50.893215 18004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a67c366512145bfb18da40e3f8e522f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:50.893461 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.873187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.10.110.2:46117)
0926 16:37:50.884127 (+ 10940us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.884133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.884154 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:37:50.884221 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.884226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.884298 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.884403 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.885299 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.886330 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.890077 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.891257 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.892680 (+ 1423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.892803 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.892857 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.893071 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10820,"thread_start_us":10714,"threads_started":1}
I20260926 16:37:50.896026 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.873339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36447 (local address 127.0.0.1:48898)
0926 16:37:50.881889 (+ 8550us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.881908 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.882005 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.889654 (+ 7649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.889658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.889683 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.889930 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.889941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.891006 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.891015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.892638 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.892652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.895313 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.895319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.895491 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.895524 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.895590 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.895723 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8435,"thread_start_us":18527,"threads_started":1}
I20260926 16:37:50.896782 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.873586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.10.110.1:36447)
0926 16:37:50.889150 (+ 15564us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.889156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.889175 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:37:50.889400 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.889407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.889490 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.890087 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.890243 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.890875 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.893938 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.895155 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.896232 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.896319 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.896370 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.896500 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15446,"thread_start_us":15630,"threads_started":1}
I20260926 16:37:50.899509 18165 tablet_bootstrap.cc:492] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Bootstrap starting.
I20260926 16:37:50.899878 17718 tablet_service.cc:1467] Processing CreateTablet for tablet 8a67c366512145bfb18da40e3f8e522f (DEFAULT_TABLE table=test-table [id=a5eb61c8cfdf4955a396b342fa7eb130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:50.900279 17718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a67c366512145bfb18da40e3f8e522f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:50.901300 18165 tablet_bootstrap.cc:654] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:50.903360 18165 tablet_bootstrap.cc:492] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: No bootstrap required, opened a new log
I20260926 16:37:50.903484 18165 ts_tablet_manager.cc:1397] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 16:37:50.904249 18165 raft_consensus.cc:348] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.904480 18165 raft_consensus.cc:374] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:50.904573 18165 raft_consensus.cc:729] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63c9ec9560440b1bb66ea08b359a00e, State: Initialized, Role: FOLLOWER
I20260926 16:37:50.904770 18165 consensus_queue.cc:260] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.905369 17863 tablet_service.cc:1467] Processing CreateTablet for tablet 8a67c366512145bfb18da40e3f8e522f (DEFAULT_TABLE table=test-table [id=a5eb61c8cfdf4955a396b342fa7eb130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:50.905745 17863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a67c366512145bfb18da40e3f8e522f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:50.907812 18167 tablet_bootstrap.cc:492] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Bootstrap starting.
I20260926 16:37:50.909664 18167 tablet_bootstrap.cc:654] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:50.910408 18169 tablet_bootstrap.cc:492] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Bootstrap starting.
I20260926 16:37:50.911979 18165 ts_tablet_manager.cc:1428] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20260926 16:37:50.913497 18167 tablet_bootstrap.cc:492] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: No bootstrap required, opened a new log
I20260926 16:37:50.913625 18167 ts_tablet_manager.cc:1397] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 16:37:50.914377 18167 raft_consensus.cc:348] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.914605 18167 raft_consensus.cc:374] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:50.914695 18167 raft_consensus.cc:729] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72159ff4d4ec48ffaa44b29765a6d720, State: Initialized, Role: FOLLOWER
I20260926 16:37:50.914896 18167 consensus_queue.cc:260] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.921515 18071 heartbeater.cc:503] Master 127.10.110.61:44281 was elected leader, sending a full tablet report...
I20260926 16:37:50.922703 18169 tablet_bootstrap.cc:654] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:50.924026 18168 raft_consensus.cc:482] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:50.924588 18168 raft_consensus.cc:504] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.926911 18168 leader_election.cc:288] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117), 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1:36447)
I20260926 16:37:50.927734 18169 tablet_bootstrap.cc:492] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: No bootstrap required, opened a new log
I20260926 16:37:50.927948 18169 ts_tablet_manager.cc:1397] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260926 16:37:50.928782 18169 raft_consensus.cc:348] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.929080 18169 raft_consensus.cc:374] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:50.929172 18169 raft_consensus.cc:729] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f56de115b446d1b5ee5054e0b038bd, State: Initialized, Role: FOLLOWER
I20260926 16:37:50.929422 18169 consensus_queue.cc:260] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.932020 18167 ts_tablet_manager.cc:1428] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260926 16:37:50.938493 17785 heartbeater.cc:503] Master 127.10.110.61:44281 was elected leader, sending a full tablet report...
I20260926 16:37:50.940052 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.912725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:49033)
0926 16:37:50.939605 (+ 26880us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.939770 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26063,"thread_start_us":26357,"threads_started":1}
I20260926 16:37:50.941455 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.932621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:37159)
0926 16:37:50.941077 (+ 8456us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.941218 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8333,"thread_start_us":8208,"threads_started":1}
I20260926 16:37:50.942387 18169 ts_tablet_manager.cc:1428] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260926 16:37:50.944260 17930 heartbeater.cc:503] Master 127.10.110.61:44281 was elected leader, sending a full tablet report...
I20260926 16:37:50.944751 18172 raft_consensus.cc:482] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:50.944959 18172 raft_consensus.cc:504] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.945477 18172 leader_election.cc:288] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193), 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:50.948700 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.942038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:53769)
0926 16:37:50.948436 (+ 6398us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.948528 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6306,"threads_started":1}
I20260926 16:37:50.949656 17883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a67c366512145bfb18da40e3f8e522f" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f56de115b446d1b5ee5054e0b038bd" is_pre_election: true
I20260926 16:37:50.949899 17883 raft_consensus.cc:2454] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 0.
I20260926 16:37:50.950271 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a67c366512145bfb18da40e3f8e522f" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63c9ec9560440b1bb66ea08b359a00e" is_pre_election: true
I20260926 16:37:50.950503 17674 leader_election.cc:302] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f56de115b446d1b5ee5054e0b038bd, 72159ff4d4ec48ffaa44b29765a6d720; no voters:
I20260926 16:37:50.950556 18024 raft_consensus.cc:2454] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 0.
I20260926 16:37:50.950881 18172 raft_consensus.cc:2781] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:50.951056 18172 raft_consensus.cc:482] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:50.951192 18172 raft_consensus.cc:3037] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:50.954134 18172 raft_consensus.cc:504] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.955276 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.939597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:40729 (local address 127.10.110.2:46117)
0926 16:37:50.939804 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.939810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.948123 (+ 8313us) server_negotiation.cc:407] Connection header received
0926 16:37:50.948321 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.948327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.948401 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.948483 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.949116 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.950195 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.952929 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.954078 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.954196 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.954264 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.954843 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.955004 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:37:50.955951 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.939468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46117 (local address 127.10.110.3:40729)
0926 16:37:50.948012 (+ 8544us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.948033 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.948176 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.948648 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.948654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.948689 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.948955 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.948967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.950366 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.950372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.952737 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.952755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.954431 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.954438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.954653 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.954688 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.955417 (+ 729us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.955573 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8414,"mutex_wait_us":704}
I20260926 16:37:50.956514 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.951056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:51031)
0926 16:37:50.956262 (+ 5206us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.956356 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5109,"threads_started":1}
I20260926 16:37:50.956789 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.940615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:42421 (local address 127.10.110.1:36447)
0926 16:37:50.940882 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.940891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.949554 (+ 8663us) server_negotiation.cc:407] Connection header received
0926 16:37:50.949780 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.949786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.949851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.949927 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:50.950680 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.951756 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.954268 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.955572 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.955708 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.955779 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.956397 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.956536 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:37:50.957422 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.940515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:36447 (local address 127.10.110.3:42421)
0926 16:37:50.949468 (+ 8953us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.949485 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:50.949642 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:50.950102 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:50.950106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:50.950129 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:50.950491 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.950506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.951945 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.951949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.954074 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:50.954091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.955976 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.955983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:50.956198 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.956233 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:37:50.956852 (+ 619us) client_negotiation.cc:240] Negotiation successful
0926 16:37:50.956995 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8846,"mutex_wait_us":1,"thread_start_us":8946,"threads_started":1}
I20260926 16:37:50.957459 18172 leader_election.cc:288] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 election: Requested vote from peers c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193), 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117)
I20260926 16:37:50.958027 17883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a67c366512145bfb18da40e3f8e522f" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f56de115b446d1b5ee5054e0b038bd"
I20260926 16:37:50.958256 17883 raft_consensus.cc:3037] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:50.958381 18024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a67c366512145bfb18da40e3f8e522f" candidate_uuid: "72159ff4d4ec48ffaa44b29765a6d720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63c9ec9560440b1bb66ea08b359a00e"
I20260926 16:37:50.958573 18024 raft_consensus.cc:3037] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:50.960608 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a67c366512145bfb18da40e3f8e522f" candidate_uuid: "c63c9ec9560440b1bb66ea08b359a00e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72159ff4d4ec48ffaa44b29765a6d720" is_pre_election: true
I20260926 16:37:50.961153 17738 raft_consensus.cc:2379] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c63c9ec9560440b1bb66ea08b359a00e in current term 1: Already voted for candidate 72159ff4d4ec48ffaa44b29765a6d720 in this term.
I20260926 16:37:50.961370 17883 raft_consensus.cc:2454] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 1.
I20260926 16:37:50.961418 18024 raft_consensus.cc:2454] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72159ff4d4ec48ffaa44b29765a6d720 in term 1.
I20260926 16:37:50.961737 17674 leader_election.cc:302] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62f56de115b446d1b5ee5054e0b038bd, 72159ff4d4ec48ffaa44b29765a6d720; no voters:
I20260926 16:37:50.962016 18172 raft_consensus.cc:2781] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:50.963272 18172 raft_consensus.cc:686] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 LEADER]: Becoming Leader. State: Replica: 72159ff4d4ec48ffaa44b29765a6d720, State: Running, Role: LEADER
I20260926 16:37:50.963567 18172 consensus_queue.cc:237] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } }
I20260926 16:37:50.964713 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.964069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:53489)
0926 16:37:50.964289 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.964498 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:37:50.965037 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.943228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:43081)
0926 16:37:50.964664 (+ 21436us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.964802 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21342,"threads_started":1}
I20260926 16:37:50.965734 17883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a67c366512145bfb18da40e3f8e522f" candidate_uuid: "c63c9ec9560440b1bb66ea08b359a00e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f56de115b446d1b5ee5054e0b038bd" is_pre_election: true
I20260926 16:37:50.966041 17883 raft_consensus.cc:2379] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c63c9ec9560440b1bb66ea08b359a00e in current term 1: Already voted for candidate 72159ff4d4ec48ffaa44b29765a6d720 in this term.
I20260926 16:37:50.966428 17960 leader_election.cc:302] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c63c9ec9560440b1bb66ea08b359a00e; no voters: 62f56de115b446d1b5ee5054e0b038bd, 72159ff4d4ec48ffaa44b29765a6d720
I20260926 16:37:50.966830 18173 raft_consensus.cc:2726] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 16:37:50.966862 17413 catalog_manager.cc:5495] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "72159ff4d4ec48ffaa44b29765a6d720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72159ff4d4ec48ffaa44b29765a6d720" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260926 16:37:50.968036 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.967680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:56673)
0926 16:37:50.967816 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.967920 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 16:37:50.993273 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.988690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.10.110.1:36447)
0926 16:37:50.988853 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:50.988873 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:50.988893 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:37:50.988989 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:50.988994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:50.989059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:50.989129 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:50.989659 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.990447 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.991696 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:50.992149 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:50.992254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:50.992616 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:50.992709 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:50.992782 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:50.992920 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 16:37:50.992991 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:37:50.996146 17883 raft_consensus.cc:1261] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Refusing update from remote peer 72159ff4d4ec48ffaa44b29765a6d720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:50.996191 18024 raft_consensus.cc:1261] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Refusing update from remote peer 72159ff4d4ec48ffaa44b29765a6d720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:50.996755 18181 consensus_queue.cc:1035] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f56de115b446d1b5ee5054e0b038bd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:50.997121 18172 consensus_queue.cc:1035] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63c9ec9560440b1bb66ea08b359a00e" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:51.000018 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.999443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:39985)
0926 16:37:50.999648 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:50.999795 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:37:51.003600 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:50.998665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:38475)
0926 16:37:51.003270 (+ 4605us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.003387 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4434}
I20260926 16:37:51.004395 18184 mvcc.cc:205] Tried to move back new op lower bound from 7333644988394532864 to 7333644988270194688. Current Snapshot: MvccSnapshot[applied={T|T < 7333644988394532864}]
I20260926 16:37:51.006817 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.006218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:56983)
0926 16:37:51.006552 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.006645 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260926 16:37:51.009680 18185 mvcc.cc:205] Tried to move back new op lower bound from 7333644988394532864 to 7333644988270194688. Current Snapshot: MvccSnapshot[applied={T|T < 7333644988394532864 or (T in {7333644988394532864})}]
I20260926 16:37:51.010375 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.005425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.2:51859)
0926 16:37:51.010093 (+ 4668us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.010215 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4543}
I20260926 16:37:51.014207 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.013651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.3:53051)
0926 16:37:51.013922 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.014018 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260926 16:37:51.015926 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.015441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.10.110.1:58971)
0926 16:37:51.015636 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.015716 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 16:37:51.016477 17414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:51.016706 17414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:51.022042 17414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2eca9fd88c084e51844c3d553f2cb5c1: Sending DeleteTablet for 3 replicas of tablet 8a67c366512145bfb18da40e3f8e522f
I20260926 16:37:51.023716 17863 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a67c366512145bfb18da40e3f8e522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:51 UTC) from {username='oryx', principal='oryx/127.10.110.61@KRBTEST.COM'} at 127.0.0.1:45364
I20260926 16:37:51.023705 17718 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a67c366512145bfb18da40e3f8e522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:51 UTC) from {username='oryx', principal='oryx/127.10.110.61@KRBTEST.COM'} at 127.0.0.1:48898
I20260926 16:37:51.024255 18004 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a67c366512145bfb18da40e3f8e522f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:51 UTC) from {username='oryx', principal='oryx/127.10.110.61@KRBTEST.COM'} at 127.0.0.1:50472
I20260926 16:37:51.024412 18193 tablet_replica.cc:330] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: stopping tablet replica
I20260926 16:37:51.024592 18193 raft_consensus.cc:2229] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:51.024659 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17577
I20260926 16:37:51.024837 18195 tablet_replica.cc:330] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: stopping tablet replica
I20260926 16:37:51.024959 18193 raft_consensus.cc:2258] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:51.025022 18195 raft_consensus.cc:2229] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:51.025288 18195 raft_consensus.cc:2258] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:51.026021 18193 ts_tablet_manager.cc:1905] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:51.028928 18195 ts_tablet_manager.cc:1905] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:51.032342 18193 ts_tablet_manager.cc:1918] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:51.032495 18193 log.cc:1196] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/8a67c366512145bfb18da40e3f8e522f
I20260926 16:37:51.032994 18193 ts_tablet_manager.cc:1939] T 8a67c366512145bfb18da40e3f8e522f P 72159ff4d4ec48ffaa44b29765a6d720: Deleting consensus metadata
I20260926 16:37:51.034049 17401 catalog_manager.cc:4841] TS 72159ff4d4ec48ffaa44b29765a6d720 (127.10.110.1:36447): tablet 8a67c366512145bfb18da40e3f8e522f (table test-table [id=a5eb61c8cfdf4955a396b342fa7eb130]) successfully deleted
I20260926 16:37:51.035228 18195 ts_tablet_manager.cc:1918] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:51.035357 18195 log.cc:1196] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/8a67c366512145bfb18da40e3f8e522f
I20260926 16:37:51.035710 18195 ts_tablet_manager.cc:1939] T 8a67c366512145bfb18da40e3f8e522f P c63c9ec9560440b1bb66ea08b359a00e: Deleting consensus metadata
I20260926 16:37:51.036634 17401 catalog_manager.cc:4841] TS c63c9ec9560440b1bb66ea08b359a00e (127.10.110.3:44193): tablet 8a67c366512145bfb18da40e3f8e522f (table test-table [id=a5eb61c8cfdf4955a396b342fa7eb130]) successfully deleted
I20260926 16:37:51.036829 18194 tablet_replica.cc:330] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: stopping tablet replica
I20260926 16:37:51.037009 18194 raft_consensus.cc:2229] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:51.037286 18194 raft_consensus.cc:2258] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:51.038103 18194 ts_tablet_manager.cc:1905] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:51.043121 18194 ts_tablet_manager.cc:1918] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:51.043232 18194 log.cc:1196] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/8a67c366512145bfb18da40e3f8e522f
I20260926 16:37:51.043630 18194 ts_tablet_manager.cc:1939] T 8a67c366512145bfb18da40e3f8e522f P 62f56de115b446d1b5ee5054e0b038bd: Deleting consensus metadata
I20260926 16:37:51.044674 17401 catalog_manager.cc:4841] TS 62f56de115b446d1b5ee5054e0b038bd (127.10.110.2:46117): tablet 8a67c366512145bfb18da40e3f8e522f (table test-table [id=a5eb61c8cfdf4955a396b342fa7eb130]) successfully deleted
I20260926 16:37:51.046182 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.026141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.10.110.1:36447)
0926 16:37:51.026296 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:51.026306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:51.026325 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:37:51.026706 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:51.026711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:51.026792 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:51.026892 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:51.028836 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:51.030070 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:51.031996 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:51.032730 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:51.032823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:51.040641 (+ 7818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:51.040657 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:51.040675 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:51.040719 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:51.043418 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:51.044281 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:51.044288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:51.044290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:51.044424 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:51.044803 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:51.044808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:51.044811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:51.045149 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:51.045492 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:51.045682 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 16:37:51.045891 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:37:51.046798 17778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:51.163432 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17577
I20260926 16:37:51.180672 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17794
I20260926 16:37:51.193262 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.181390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.10.110.2:46117)
0926 16:37:51.181556 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:51.181562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:51.181582 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:37:51.181767 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:51.181773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:51.181853 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:51.181939 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:51.183133 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:51.183969 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:51.185145 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:51.185541 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:51.185643 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:51.188057 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:51.188072 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:51.188090 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:51.188122 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:51.190748 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:51.191389 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:51.191396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:51.191398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:51.191517 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:51.191878 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:51.191883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:51.191885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:51.192229 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:51.192513 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:51.192757 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 16:37:51.192954 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:51.193830 17923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:51.284409 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17794
I20260926 16:37:51.297964 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17941
I20260926 16:37:51.310385 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.298730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.10.110.3:44193)
0926 16:37:51.298890 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:51.298895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:51.298913 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:37:51.299060 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:51.299063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:51.299139 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:51.299221 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:51.300674 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:51.301366 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:51.302538 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:51.302947 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:51.303036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:51.305420 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:51.305435 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:51.305452 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:51.305483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:51.308228 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:51.308829 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:51.308835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:51.308838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:51.308961 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:51.309237 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:51.309240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:51.309242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:51.309566 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:51.309770 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:51.309992 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 16:37:51.310148 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:37:51.311096 18064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:51.340500 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.340039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42428)
0926 16:37:51.340207 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.340309 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:37:51.341415 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:51.341123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34111 (local address 127.0.0.1:42432)
0926 16:37:51.341242 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:51.341307 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 16:37:51.406569 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17941
I20260926 16:37:51.423799 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17278
I20260926 16:37:51.424765 17444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:51.543953 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17278
I20260926 16:37:51.560739 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 17378
I20260926 16:37:51.561781 17635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:51.674031 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 17378
2026-09-26T16:37:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8828 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:51 dist-test-slave-whjp krb5kdc[18204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:51 dist-test-slave-whjp krb5kdc[18204](info): set up 2 sockets
Sep 26 16:37:51 dist-test-slave-whjp krb5kdc[18204](info): commencing operation
krb5kdc: starting...
W20260926 16:37:52.677235 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:52 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab.
Entry for principal oryx/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab.
I20260926 16:37:52.876974 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33365
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab
--principal=oryx/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:53.193233 18217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:53.193522 18217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:53.193635 18217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:53.200956 18217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:53.201071 18217 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:37:53.201131 18217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:53.201216 18217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:53.201279 18217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:53.201365 18217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:53.211849 18217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33365
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.62.keytab
--principal=oryx/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:53.213194 18217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:53.214556 18217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:53.225664 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:53.225600 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:53.225638 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:53.226636 18217 server_base.cc:1034] running on GCE node
I20260926 16:37:53.227404 18217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:53.228819 18217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:53.230041 18217 hybrid_clock.cc:648] HybridClock initialized: now 1790440673229977 us; error 38 us; skew 500 ppm
I20260926 16:37:53.230351 18217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:53 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440673, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:53.233290 18217 init.cc:377] Logged in from keytab as oryx/127.10.110.62@KRBTEST.COM (short username oryx)
I20260926 16:37:53.234619 18217 webserver.cc:460] Webserver started at http://127.10.110.62:39483/ using document root <none> and password file <none>
I20260926 16:37:53.235301 18217 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:53.235383 18217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:53.235667 18217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:53.238574 18217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "f0c7df1aef4a4add8e05a89ae823a931"
format_stamp: "Formatted at 2026-09-26 16:37:53 on dist-test-slave-whjp"
server_key: "578c03517e2f1409a5157dba70f03800"
server_key_iv: "2b154b6f3e6fb2204d8ad6594a50e56d"
server_key_version: "encryptionkey@0"
I20260926 16:37:53.239359 18217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "f0c7df1aef4a4add8e05a89ae823a931"
format_stamp: "Formatted at 2026-09-26 16:37:53 on dist-test-slave-whjp"
server_key: "578c03517e2f1409a5157dba70f03800"
server_key_iv: "2b154b6f3e6fb2204d8ad6594a50e56d"
server_key_version: "encryptionkey@0"
I20260926 16:37:53.244367 18217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 16:37:53.247784 18232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:53.249118 18217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:37:53.249255 18217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "f0c7df1aef4a4add8e05a89ae823a931"
format_stamp: "Formatted at 2026-09-26 16:37:53 on dist-test-slave-whjp"
server_key: "578c03517e2f1409a5157dba70f03800"
server_key_iv: "2b154b6f3e6fb2204d8ad6594a50e56d"
server_key_version: "encryptionkey@0"
I20260926 16:37:53.249475 18217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:53.301952 18217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:53.306212 18217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:53.306474 18217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:53.326861 18217 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33365
I20260926 16:37:53.326901 18293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33365 every 8 connection(s)
I20260926 16:37:53.328184 18217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:37:53.331753 18294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:53.336886 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 18217
I20260926 16:37:53.337072 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:37:53.337807 10680 external_mini_cluster.cc:1428] Setting key 7da6297b54053e238f3f57905ada122a
I20260926 16:37:53.342667 18294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Bootstrap starting.
I20260926 16:37:53.346233 18294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:53.347503 18294 log.cc:824] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:53.350914 18294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: No bootstrap required, opened a new log
Sep 26 16:37:53 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:53.359695 18294 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c7df1aef4a4add8e05a89ae823a931" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33365 } }
I20260926 16:37:53.360090 18294 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:53.360179 18294 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c7df1aef4a4add8e05a89ae823a931, State: Initialized, Role: FOLLOWER
I20260926 16:37:53.360658 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:53.339794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.10.110.62:33365)
0926 16:37:53.340624 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:53.340639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:53.340677 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:37:53.342281 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:53.342302 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:53.342683 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:53.343010 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:53.344738 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.346066 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.347725 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.348779 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.348938 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:53.353519 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:53.353553 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:53.353577 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:53.353632 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:53.356658 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:53.357414 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:53.357420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:53.357427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:53.357618 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:53.358012 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:53.358015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:53.358018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:53.358543 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:53.358772 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:53.359173 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:37:53.359549 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":342,"threads_started":1}
I20260926 16:37:53.360787 18294 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c7df1aef4a4add8e05a89ae823a931" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33365 } }
I20260926 16:37:53.361033 18294 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:37:53.361166 18294 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:37:53.361327 18294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:53.363523 18294 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c7df1aef4a4add8e05a89ae823a931" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33365 } }
I20260926 16:37:53.364059 18294 leader_election.cc:302] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0c7df1aef4a4add8e05a89ae823a931; no voters:
I20260926 16:37:53.364598 18294 leader_election.cc:288] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:37:53.364729 18299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:53.365361 18299 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [term 1 LEADER]: Becoming Leader. State: Replica: f0c7df1aef4a4add8e05a89ae823a931, State: Running, Role: LEADER
I20260926 16:37:53.365952 18299 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c7df1aef4a4add8e05a89ae823a931" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33365 } }
I20260926 16:37:53.366614 18294 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:37:53.368593 18301 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0c7df1aef4a4add8e05a89ae823a931. Latest consensus state: current_term: 1 leader_uuid: "f0c7df1aef4a4add8e05a89ae823a931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c7df1aef4a4add8e05a89ae823a931" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33365 } } }
I20260926 16:37:53.368975 18301 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:53.369750 18300 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0c7df1aef4a4add8e05a89ae823a931" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0c7df1aef4a4add8e05a89ae823a931" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33365 } } }
I20260926 16:37:53.370159 18300 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931 [sys.catalog]: This master's current role is: LEADER
I20260926 16:37:53.370404 18306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:37:53.375165 18306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:37:53.385003 18306 catalog_manager.cc:1347] Generated new cluster ID: 217b136965a84df8ad389ceec347cc39
I20260926 16:37:53.385128 18306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:37:53.409137 18306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:37:53.410458 18306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:37:53.416967 18306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Generated new TSK 0
I20260926 16:37:53.417738 18306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab.
Entry for principal oryx/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab.
I20260926 16:37:53.476099 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33365
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:53.797289 18320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:53.797540 18320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:53.797632 18320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:53.804724 18320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:53.804828 18320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:53.804924 18320 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:53.805003 18320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:37:53.814637 18320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:53.815822 18320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:53.817088 18320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:53.829658 18325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:53.829622 18328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:53.829691 18326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:53.830719 18320 server_base.cc:1034] running on GCE node
I20260926 16:37:53.831326 18320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:53.832319 18320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:53.833554 18320 hybrid_clock.cc:648] HybridClock initialized: now 1790440673833473 us; error 59 us; skew 500 ppm
I20260926 16:37:53.833882 18320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:53 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440673, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:53.836884 18320 init.cc:377] Logged in from keytab as oryx/127.10.110.1@KRBTEST.COM (short username oryx)
I20260926 16:37:53.838181 18320 webserver.cc:460] Webserver started at http://127.10.110.1:36091/ using document root <none> and password file <none>
I20260926 16:37:53.838829 18320 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:53.838913 18320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:53.839183 18320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:53.842049 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "5a4499d7f5214491a342137c477bc0e2"
format_stamp: "Formatted at 2026-09-26 16:37:53 on dist-test-slave-whjp"
server_key: "11ef5b5ae66cb1a2a2828a9eb51ddeed"
server_key_iv: "b84e53985995d51cab58da4c2dbf9f43"
server_key_version: "encryptionkey@0"
I20260926 16:37:53.842783 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4499d7f5214491a342137c477bc0e2"
format_stamp: "Formatted at 2026-09-26 16:37:53 on dist-test-slave-whjp"
server_key: "11ef5b5ae66cb1a2a2828a9eb51ddeed"
server_key_iv: "b84e53985995d51cab58da4c2dbf9f43"
server_key_version: "encryptionkey@0"
I20260926 16:37:53.847759 18320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 16:37:53.851117 18335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:53.852435 18320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:37:53.852591 18320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "5a4499d7f5214491a342137c477bc0e2"
format_stamp: "Formatted at 2026-09-26 16:37:53 on dist-test-slave-whjp"
server_key: "11ef5b5ae66cb1a2a2828a9eb51ddeed"
server_key_iv: "b84e53985995d51cab58da4c2dbf9f43"
server_key_version: "encryptionkey@0"
I20260926 16:37:53.852784 18320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:53.878528 18320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:53.882833 18320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:53.883054 18320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:53.885677 18320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:53.885787 18320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:53.885921 18320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:53.885984 18320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:53 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440673, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:53.910215 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:53.886349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:60135 (local address 127.10.110.62:33365)
0926 16:37:53.888028 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:53.888034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:53.888200 (+ 166us) server_negotiation.cc:407] Connection header received
0926 16:37:53.890891 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:53.890898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:53.890992 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:53.891117 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:53.893454 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.894522 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.897380 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.897997 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.898132 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:53.903341 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:53.903348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:53.903353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:53.903407 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:53.906406 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:53.907347 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:53.907354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:53.907359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:53.907523 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:53.908120 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:53.908125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:53.908129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:53.908420 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:53.908551 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:53.909633 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0926 16:37:53.909851 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":1488,"threads_started":1}
I20260926 16:37:53.911770 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:53.887067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.10.110.1:60135)
0926 16:37:53.888068 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:53.888108 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:53.890557 (+ 2449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:53.891395 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:53.891409 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:53.892284 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:53.893248 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:53.893278 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.894706 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.894712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:53.897211 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:53.897228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.898266 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.898273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:53.898597 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:53.899541 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:53.899575 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:53.903053 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:53.906555 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:53.906565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:53.906586 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:53.907180 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:53.907654 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:53.907659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:53.907663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:53.907989 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:53.908544 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:53.908552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:53.909187 (+ 635us) client_negotiation.cc:769] Sending connection context
0926 16:37:53.909532 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 16:37:53.910157 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":388,"threads_started":1}
I20260926 16:37:53.922950 18320 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:41163
I20260926 16:37:53.922972 18455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:41163 every 8 connection(s)
I20260926 16:37:53.924588 18320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:37:53.926779 18456 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33365
I20260926 16:37:53.927016 18456 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:53.927772 18456 heartbeater.cc:511] Master 127.10.110.62:33365 requested a full tablet report, sending...
I20260926 16:37:53.928066 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 18320
I20260926 16:37:53.928263 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:37:53.928998 10680 external_mini_cluster.cc:1428] Setting key 3bc57170cc469b8888a8a0b49f37f4c7
I20260926 16:37:53.930150 18249 ts_manager.cc:194] Registered new tserver with Master: 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163)
I20260926 16:37:53.931569 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:53.918618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:44077 (local address 127.10.110.62:33365)
0926 16:37:53.918785 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:53.918791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:53.919351 (+ 560us) server_negotiation.cc:407] Connection header received
0926 16:37:53.919542 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:53.919548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:53.919627 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:53.919720 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:53.921203 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.921909 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.923433 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.924052 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.924189 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:53.926201 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:53.926208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:53.926213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:53.926252 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:53.928904 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:53.929541 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:53.929547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:53.929551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:53.929722 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:53.930116 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:53.930122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:53.930125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:53.930471 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:53.930594 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:53.931136 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 16:37:53.931349 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:37:53.931881 18249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:60135
I20260926 16:37:53.932241 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:53.918684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.10.110.1:44077)
0926 16:37:53.919251 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:53.919266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:53.919415 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:53.919778 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:53.919783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:53.920590 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:53.921055 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:53.921068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.922085 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.922091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:53.923300 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:53.923312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:53.924227 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:53.924231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:53.924353 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:53.924945 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:53.924962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:53.926046 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:53.929053 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:53.929059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:53.929063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:53.929437 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:53.929844 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:53.929849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:53.929853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:53.929989 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:53.930686 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:53.930690 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:53.930867 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 16:37:53.931685 (+ 818us) client_negotiation.cc:240] Negotiation successful
0926 16:37:53.931965 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
WARNING: no policy specified for oryx/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab.
Entry for principal oryx/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab.
I20260926 16:37:53.973121 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33365
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:54.284822 18460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:54.285108 18460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:54.285202 18460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:54.292228 18460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:54.292332 18460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:54.292415 18460 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:54.292495 18460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:37:54.302384 18460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:54.303571 18460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:54.304836 18460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:54.317056 18465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:54.317026 18466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:54.317076 18468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:54.317795 18460 server_base.cc:1034] running on GCE node
I20260926 16:37:54.318334 18460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:54.319155 18460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:54.320369 18460 hybrid_clock.cc:648] HybridClock initialized: now 1790440674320302 us; error 46 us; skew 500 ppm
I20260926 16:37:54.320636 18460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440674, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:54.323496 18460 init.cc:377] Logged in from keytab as oryx/127.10.110.2@KRBTEST.COM (short username oryx)
I20260926 16:37:54.324831 18460 webserver.cc:460] Webserver started at http://127.10.110.2:38615/ using document root <none> and password file <none>
I20260926 16:37:54.325482 18460 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:54.325575 18460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:54.325879 18460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:54.328711 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "39b8d2513b03440f81a14bb87492fbb3"
format_stamp: "Formatted at 2026-09-26 16:37:54 on dist-test-slave-whjp"
server_key: "162b28ba0b5b4daeaff11ebe5adfc32b"
server_key_iv: "d18c7f12f2352714101160c6725a412d"
server_key_version: "encryptionkey@0"
I20260926 16:37:54.329459 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "39b8d2513b03440f81a14bb87492fbb3"
format_stamp: "Formatted at 2026-09-26 16:37:54 on dist-test-slave-whjp"
server_key: "162b28ba0b5b4daeaff11ebe5adfc32b"
server_key_iv: "d18c7f12f2352714101160c6725a412d"
server_key_version: "encryptionkey@0"
I20260926 16:37:54.334234 18460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:37:54.337689 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:54.338989 18460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:37:54.339129 18460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "39b8d2513b03440f81a14bb87492fbb3"
format_stamp: "Formatted at 2026-09-26 16:37:54 on dist-test-slave-whjp"
server_key: "162b28ba0b5b4daeaff11ebe5adfc32b"
server_key_iv: "d18c7f12f2352714101160c6725a412d"
server_key_version: "encryptionkey@0"
I20260926 16:37:54.339337 18460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:54.359797 18460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:54.363749 18460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:54.364018 18460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:54.366763 18460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:54.366850 18460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:54.366984 18460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:54.367039 18460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440674, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:54.391522 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.367484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:50137 (local address 127.10.110.62:33365)
0926 16:37:54.367656 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:54.367662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:54.369290 (+ 1628us) server_negotiation.cc:407] Connection header received
0926 16:37:54.372253 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:54.372284 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:54.372461 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:54.372605 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:54.375806 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.377129 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.380067 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.380643 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.380778 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.385133 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:54.385138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:54.385141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:54.385173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:54.388339 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.389284 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.389291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.389295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.389412 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.389825 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.389830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.389834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.390123 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:54.390261 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:54.390974 (+ 713us) server_negotiation.cc:299] Negotiation successful
0926 16:37:54.391198 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:37:54.392930 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.368141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.10.110.2:50137)
0926 16:37:54.369021 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:54.369113 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:54.371899 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:54.372864 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:54.372877 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:54.373648 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:54.374560 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.374592 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.377436 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.377443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.379902 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.379918 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.380925 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.380934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.381250 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.382057 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:54.382088 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:54.384830 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:54.388480 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.388495 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.388515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.389160 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.389510 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.389515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.389519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.389717 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.390241 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:54.390249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:54.390613 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 16:37:54.390856 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 16:37:54.391360 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":330,"threads_started":1}
I20260926 16:37:54.410384 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:36321 every 8 connection(s)
I20260926 16:37:54.410411 18460 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:36321
I20260926 16:37:54.411887 18460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:37:54.412243 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 18460
I20260926 16:37:54.412421 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:37:54.412957 10680 external_mini_cluster.cc:1428] Setting key 3c0102902171678485db349470f5e901
I20260926 16:37:54.413672 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.399601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.10.110.2:46361)
0926 16:37:54.400255 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:54.400269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:54.400396 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:54.400860 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:54.400866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:54.401412 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:54.401731 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.401740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.402778 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.402783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.404452 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.404467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.405504 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.405512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.405692 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.406374 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:54.406399 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:54.407443 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:54.410957 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.410960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.410963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.411322 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.411693 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.411696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.411699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.411806 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.412599 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:54.412604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:54.412803 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 16:37:54.412932 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 16:37:54.413251 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":396,"threads_started":1}
I20260926 16:37:54.414443 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.399408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:46361 (local address 127.10.110.62:33365)
0926 16:37:54.399598 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:54.399605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:54.400345 (+ 740us) server_negotiation.cc:407] Connection header received
0926 16:37:54.400516 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:54.400521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:54.400594 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:54.400687 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:54.401860 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.402660 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.404636 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.405171 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.405287 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.407616 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:54.407621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:54.407624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:54.407652 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:54.410849 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.411436 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.411441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.411444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.411588 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.411965 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.411970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.411973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.412398 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:54.412533 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:54.413952 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0926 16:37:54.414158 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:37:54.415241 18595 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33365
I20260926 16:37:54.415506 18595 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:54.416447 18595 heartbeater.cc:511] Master 127.10.110.62:33365 requested a full tablet report, sending...
I20260926 16:37:54.417984 18248 ts_manager.cc:194] Registered new tserver with Master: 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321)
I20260926 16:37:54.419132 18248 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:50137
WARNING: no policy specified for oryx/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab.
Entry for principal oryx/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab.
I20260926 16:37:54.458906 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33365
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:54.767006 18599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:54.767288 18599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:54.767392 18599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:54.774652 18599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:54.774770 18599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:54.774852 18599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:37:54.774930 18599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:37:54.784659 18599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.110.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:54.785864 18599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:54.787129 18599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:54.799489 18604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:54.799489 18607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:54.799681 18605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:54.800297 18599 server_base.cc:1034] running on GCE node
I20260926 16:37:54.800841 18599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:54.801708 18599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:54.802922 18599 hybrid_clock.cc:648] HybridClock initialized: now 1790440674802856 us; error 42 us; skew 500 ppm
I20260926 16:37:54.803190 18599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440674, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:37:54.806105 18599 init.cc:377] Logged in from keytab as oryx/127.10.110.3@KRBTEST.COM (short username oryx)
I20260926 16:37:54.807497 18599 webserver.cc:460] Webserver started at http://127.10.110.3:35529/ using document root <none> and password file <none>
I20260926 16:37:54.808195 18599 fs_manager.cc:359] Metadata directory not provided
I20260926 16:37:54.808291 18599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:37:54.808583 18599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:37:54.811465 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
format_stamp: "Formatted at 2026-09-26 16:37:54 on dist-test-slave-whjp"
server_key: "7f4ec90862483660bc2d315897a88e00"
server_key_iv: "3b2557a39999b46d97f4f71b4c531b76"
server_key_version: "encryptionkey@0"
I20260926 16:37:54.812219 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
format_stamp: "Formatted at 2026-09-26 16:37:54 on dist-test-slave-whjp"
server_key: "7f4ec90862483660bc2d315897a88e00"
server_key_iv: "3b2557a39999b46d97f4f71b4c531b76"
server_key_version: "encryptionkey@0"
I20260926 16:37:54.817147 18599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:37:54.820464 18614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:54.821972 18599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:37:54.822116 18599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
format_stamp: "Formatted at 2026-09-26 16:37:54 on dist-test-slave-whjp"
server_key: "7f4ec90862483660bc2d315897a88e00"
server_key_iv: "3b2557a39999b46d97f4f71b4c531b76"
server_key_version: "encryptionkey@0"
I20260926 16:37:54.822324 18599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:37:54.840209 18599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:37:54.844506 18599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:37:54.844776 18599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:37:54.847345 18599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:37:54.847438 18599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:37:54.847575 18599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:37:54.847630 18599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440674, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:54.870975 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.848158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:34513 (local address 127.10.110.62:33365)
0926 16:37:54.848342 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:54.848348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:54.849738 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 16:37:54.852608 (+ 2870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:54.852614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:54.852701 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:54.852825 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:54.855365 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.856279 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.859233 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.859691 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.859795 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.864593 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:54.864600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:54.864604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:54.864639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:54.867654 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.868473 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.868479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.868483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.868635 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.869203 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.869207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.869209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.869489 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:54.869617 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:54.870380 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 16:37:54.870608 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:37:54.872144 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.848708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.10.110.3:34513)
0926 16:37:54.849551 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:54.849601 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:54.852234 (+ 2633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:54.853124 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:54.853136 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:54.854101 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:54.855145 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.855177 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.856433 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.856440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.859057 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.859084 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.859987 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.859995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.860319 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.861215 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:54.861251 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:54.864370 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:54.867802 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.867811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.867847 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.868356 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.868847 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.868852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.868856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.869072 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.869609 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:54.869617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:54.869993 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 16:37:54.870231 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 16:37:54.870803 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":331,"threads_started":1}
I20260926 16:37:54.894428 18599 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:44383
I20260926 16:37:54.894441 18733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:44383 every 8 connection(s)
I20260926 16:37:54.896041 18599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:37:54.896333 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.882288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:32991 (local address 127.10.110.62:33365)
0926 16:37:54.882505 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:54.882512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:54.883329 (+ 817us) server_negotiation.cc:407] Connection header received
0926 16:37:54.883472 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:54.883478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:54.883564 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:54.883674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:54.884755 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.885837 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.887689 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.888346 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.888487 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.890052 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:54.890059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:54.890064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:54.890099 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:54.893452 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.894475 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.894479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.894483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.894625 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:54.894954 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:54.894959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:54.894962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:54.895293 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:54.895449 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:54.895783 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 16:37:54.896041 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:37:54.896353 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:54.882303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.10.110.3:32991)
0926 16:37:54.883220 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:54.883239 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:54.883371 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:54.883725 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:54.883734 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:54.884282 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:54.884602 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.884615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.886027 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.886032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.887543 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:54.887560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.888514 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.888519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:54.888625 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:54.889186 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:54.889202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:54.889866 (+ 664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:54.893618 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.893625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.893630 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.894301 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.894725 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:54.894730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:54.894734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:54.894857 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:54.895398 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:54.895402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:54.895553 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 16:37:54.895667 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 16:37:54.895938 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":639,"threads_started":1}
I20260926 16:37:54.897415 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 18599
I20260926 16:37:54.897591 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:37:54.898162 10680 external_mini_cluster.cc:1428] Setting key 5564e32248621c4a96071b72bd82a42a
I20260926 16:37:54.898597 18734 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33365
I20260926 16:37:54.898823 18734 heartbeater.cc:464] Registering TS with master...
I20260926 16:37:54.899601 18734 heartbeater.cc:511] Master 127.10.110.62:33365 requested a full tablet report, sending...
I20260926 16:37:54.901090 18248 ts_manager.cc:194] Registered new tserver with Master: 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:54.902149 18248 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:34513
I20260926 16:37:54.902251 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 16:37:54.935160 18456 heartbeater.cc:503] Master 127.10.110.62:33365 was elected leader, sending a full tablet report...
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:37:54 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:37:55.421449 18595 heartbeater.cc:503] Master 127.10.110.62:33365 was elected leader, sending a full tablet report...
I20260926 16:37:55.904506 18734 heartbeater.cc:503] Master 127.10.110.62:33365 was elected leader, sending a full tablet report...
W20260926 16:37:55.909493 18742 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.964s user 0.005s sys 0.008s
W20260926 16:37:55.910547 18344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:54.940329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.10.110.62:33365)
0926 16:37:54.940531 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:54.940537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:54.940785 (+ 248us) server_negotiation.cc:407] Connection header received
0926 16:37:54.940945 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:54.940952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:54.941012 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:54.941090 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:54.942443 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.943178 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.944466 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:54.944882 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:54.944988 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.910293 (+965305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51156: BlockingRecv error: recv got EOF from 127.0.0.1:51156 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 16:37:55 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:55.928107 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.913945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.10.110.62:33365)
0926 16:37:55.914113 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:55.914118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:55.914513 (+ 395us) server_negotiation.cc:407] Connection header received
0926 16:37:55.914682 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:55.914688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:55.914768 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:55.914849 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:55.916153 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.916899 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.918090 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.918503 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.918616 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.922614 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:55.922620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:55.922624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:55.922664 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:55.925317 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:55.925992 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:55.925999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:55.926001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:55.926227 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:55.926606 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:55.926612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:55.926615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:55.926923 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:55.927233 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:55.927524 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 16:37:55.927733 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:37:55.932152 18248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:55.938954 18248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:55.962472 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.950779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.10.110.1:41163)
0926 16:37:55.951313 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:55.951322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:55.952312 (+ 990us) server_negotiation.cc:407] Connection header received
0926 16:37:55.952420 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:55.952427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:55.952668 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:55.952858 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:55.953762 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.955340 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.958110 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.960260 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.960423 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.960668 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:55.961636 (+ 968us) server_negotiation.cc:299] Negotiation successful
0926 16:37:55.961799 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":228,"threads_started":1}
I20260926 16:37:55.962662 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.950617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41163 (local address 127.0.0.1:43252)
0926 16:37:55.951908 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:55.951971 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:55.952139 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:55.953104 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:55.953112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:55.953144 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:55.953548 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:55.953560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.955505 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.955511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:55.957862 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:55.957877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.960803 (+ 2926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.960821 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:55.961050 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.961105 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 16:37:55.961243 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 16:37:55.961437 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"spinlock_wait_cycles":7808,"thread_start_us":833,"threads_started":1}
I20260926 16:37:55.967125 18386 tablet_service.cc:1467] Processing CreateTablet for tablet 51215bb2bf44411d917e62337c24350b (DEFAULT_TABLE table=test-table [id=2721bf5b0e04400c97e824896c466e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:55.968816 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51215bb2bf44411d917e62337c24350b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:55.968871 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.951614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.10.110.2:36321)
0926 16:37:55.961796 (+ 10182us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:55.961804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:55.961835 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:37:55.961934 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:55.961942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:55.962216 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:55.962428 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:55.963250 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.964574 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.966467 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.967934 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.968055 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.968260 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:55.968450 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 16:37:55.968599 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9251,"thread_start_us":9132,"threads_started":1}
I20260926 16:37:55.968905 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.951098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:36321 (local address 127.0.0.1:53256)
0926 16:37:55.951908 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:55.951971 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:55.952139 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:55.962640 (+ 10501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:55.962648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:55.962673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:55.963062 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:55.963074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.964693 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.964698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:55.966243 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:55.966257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.968129 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.968135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:55.968284 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.968323 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:37:55.968398 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 16:37:55.968527 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":45,"spinlock_wait_cycles":7296,"thread_start_us":1364,"threads_started":1}
I20260926 16:37:55.969833 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.952335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44383 (local address 127.0.0.1:52732)
0926 16:37:55.962905 (+ 10570us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:55.962928 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:55.963131 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:55.963888 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:55.963893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:55.963916 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:55.964057 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:55.964064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.965705 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.965711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:55.967560 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:55.967575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.969277 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.969281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:55.969442 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.969464 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:37:55.969531 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 16:37:55.969623 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10464,"mutex_wait_us":20,"thread_start_us":10820,"threads_started":1}
I20260926 16:37:55.970014 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:55.952851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.10.110.3:44383)
0926 16:37:55.961398 (+ 8547us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:55.961407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:55.963023 (+ 1616us) server_negotiation.cc:407] Connection header received
0926 16:37:55.963331 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:55.963338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:55.963570 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:55.963685 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:55.964182 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.965555 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.967713 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:55.969135 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:55.969282 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:55.969468 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:55.969565 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 16:37:55.969724 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8373,"thread_start_us":8703,"threads_started":1}
I20260926 16:37:55.972651 18525 tablet_service.cc:1467] Processing CreateTablet for tablet 51215bb2bf44411d917e62337c24350b (DEFAULT_TABLE table=test-table [id=2721bf5b0e04400c97e824896c466e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:55.973258 18664 tablet_service.cc:1467] Processing CreateTablet for tablet 51215bb2bf44411d917e62337c24350b (DEFAULT_TABLE table=test-table [id=2721bf5b0e04400c97e824896c466e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:55.974140 18525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51215bb2bf44411d917e62337c24350b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:55.974664 18664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51215bb2bf44411d917e62337c24350b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:55.984506 18759 tablet_bootstrap.cc:492] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Bootstrap starting.
I20260926 16:37:55.987548 18759 tablet_bootstrap.cc:654] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:55.988380 18761 tablet_bootstrap.cc:492] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Bootstrap starting.
I20260926 16:37:55.988857 18759 log.cc:824] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:55.989207 18760 tablet_bootstrap.cc:492] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Bootstrap starting.
I20260926 16:37:55.992206 18759 tablet_bootstrap.cc:492] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: No bootstrap required, opened a new log
I20260926 16:37:55.992461 18759 ts_tablet_manager.cc:1397] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 16:37:55.992833 18760 tablet_bootstrap.cc:654] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:55.993176 18761 tablet_bootstrap.cc:654] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:55.994118 18760 log.cc:824] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:55.994269 18761 log.cc:824] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Log is configured to *not* fsync() on all Append() calls
I20260926 16:37:55.997046 18760 tablet_bootstrap.cc:492] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: No bootstrap required, opened a new log
I20260926 16:37:55.997296 18761 tablet_bootstrap.cc:492] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: No bootstrap required, opened a new log
I20260926 16:37:55.997298 18760 ts_tablet_manager.cc:1397] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 16:37:55.997550 18761 ts_tablet_manager.cc:1397] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 16:37:56.001961 18759 raft_consensus.cc:348] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.002435 18759 raft_consensus.cc:374] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.002580 18759 raft_consensus.cc:729] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.003262 18759 consensus_queue.cc:260] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.005242 18759 ts_tablet_manager.cc:1428] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260926 16:37:56.005174 18760 raft_consensus.cc:348] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.005672 18760 raft_consensus.cc:374] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.005776 18760 raft_consensus.cc:729] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.006430 18760 consensus_queue.cc:260] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.007193 18761 raft_consensus.cc:348] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.007658 18761 raft_consensus.cc:374] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.007786 18761 raft_consensus.cc:729] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894b1a87ba9f4f8c92e7025a28a46b89, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.008201 18760 ts_tablet_manager.cc:1428] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 16:37:56.008529 18761 consensus_queue.cc:260] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.010210 18761 ts_tablet_manager.cc:1428] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260926 16:37:56.154395 18735 tablet.cc:2369] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:37:56.170862 18596 tablet.cc:2369] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:37:56.177666 18457 tablet.cc:2369] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:37:56.214906 18765 raft_consensus.cc:482] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:56.215209 18765 raft_consensus.cc:504] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.216846 18765 leader_election.cc:288] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:56.222784 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.217370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:44439 (local address 127.10.110.2:36321)
0926 16:37:56.217571 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.217576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.217824 (+ 248us) server_negotiation.cc:407] Connection header received
0926 16:37:56.218036 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.218040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.218095 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.218168 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:56.218657 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.219348 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.221215 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.221958 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.222106 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.222180 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.222439 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.222559 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:37:56.222923 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.217091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:36321 (local address 127.10.110.1:44439)
0926 16:37:56.217711 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.217733 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.217886 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.218207 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.218212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.218237 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:56.218529 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.218538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.219470 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.219474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.221055 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.221068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.222145 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.222151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.222294 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.222317 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.222394 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.222523 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":4,"thread_start_us":318,"threads_started":1}
I20260926 16:37:56.223191 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.217252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:46049 (local address 127.10.110.3:44383)
0926 16:37:56.217436 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.217443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.218300 (+ 857us) server_negotiation.cc:407] Connection header received
0926 16:37:56.218491 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.218495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.218549 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.218621 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:56.219124 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.219801 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.221584 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.222338 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.222486 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.222556 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.222821 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.222941 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:37:56.223397 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.217104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44383 (local address 127.10.110.1:46049)
0926 16:37:56.218187 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.218203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.218349 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.218649 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.218653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.218697 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:56.218972 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.218983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.219956 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.219962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.221433 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.221445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.222542 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.222548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.222683 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.222704 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.222778 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.222899 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":538,"thread_start_us":272,"threads_started":1}
I20260926 16:37:56.223996 18545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51215bb2bf44411d917e62337c24350b" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b8d2513b03440f81a14bb87492fbb3" is_pre_election: true
I20260926 16:37:56.224351 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51215bb2bf44411d917e62337c24350b" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" is_pre_election: true
I20260926 16:37:56.224602 18545 raft_consensus.cc:2454] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 0.
I20260926 16:37:56.224843 18684 raft_consensus.cc:2454] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 0.
I20260926 16:37:56.225263 18337 leader_election.cc:302] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 5a4499d7f5214491a342137c477bc0e2; no voters:
I20260926 16:37:56.225642 18765 raft_consensus.cc:2781] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:56.225796 18765 raft_consensus.cc:482] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:56.225898 18765 raft_consensus.cc:3037] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.228087 18765 raft_consensus.cc:504] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.228700 18765 leader_election.cc:288] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 election: Requested vote from peers 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:56.229161 18545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51215bb2bf44411d917e62337c24350b" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b8d2513b03440f81a14bb87492fbb3"
I20260926 16:37:56.229249 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51215bb2bf44411d917e62337c24350b" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
I20260926 16:37:56.229338 18545 raft_consensus.cc:3037] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.229463 18684 raft_consensus.cc:3037] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.231426 18545 raft_consensus.cc:2454] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 1.
I20260926 16:37:56.231766 18684 raft_consensus.cc:2454] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 1.
I20260926 16:37:56.231810 18337 leader_election.cc:302] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 5a4499d7f5214491a342137c477bc0e2; no voters:
I20260926 16:37:56.232137 18765 raft_consensus.cc:2781] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:56.232570 18765 raft_consensus.cc:686] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Running, Role: LEADER
I20260926 16:37:56.233023 18765 consensus_queue.cc:237] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.237272 18248 catalog_manager.cc:5495] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "5a4499d7f5214491a342137c477bc0e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:56.255383 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.250347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.10.110.1:41163)
0926 16:37:56.250509 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.250514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.250531 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:37:56.250614 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.250617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.250696 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.250814 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:56.251149 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.251863 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.253411 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.253870 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.253986 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.254406 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:56.254933 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:56.255035 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.255076 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.255143 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:37:56.263516 18545 raft_consensus.cc:1261] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Refusing update from remote peer 5a4499d7f5214491a342137c477bc0e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:56.263486 18684 raft_consensus.cc:1261] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Refusing update from remote peer 5a4499d7f5214491a342137c477bc0e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:56.264613 18770 consensus_queue.cc:1035] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:56.264969 18765 consensus_queue.cc:1035] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:56.280073 18773 mvcc.cc:205] Tried to move back new op lower bound from 7333645009965338624 to 7333645009853689856. Current Snapshot: MvccSnapshot[applied={T|T < 7333645009965338624}]
I20260926 16:37:56.281462 18775 mvcc.cc:205] Tried to move back new op lower bound from 7333645009965338624 to 7333645009853689856. Current Snapshot: MvccSnapshot[applied={T|T < 7333645009965338624 or (T in {7333645009965338624})}]
I20260926 16:37:56.293329 18249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:56.293545 18249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:56.297129 18249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Sending DeleteTablet for 3 replicas of tablet 51215bb2bf44411d917e62337c24350b
I20260926 16:37:56.298410 18525 tablet_service.cc:1514] Processing DeleteTablet for tablet 51215bb2bf44411d917e62337c24350b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:56 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53256
I20260926 16:37:56.298537 18386 tablet_service.cc:1514] Processing DeleteTablet for tablet 51215bb2bf44411d917e62337c24350b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:56 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 16:37:56.299121 18664 tablet_service.cc:1514] Processing DeleteTablet for tablet 51215bb2bf44411d917e62337c24350b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:56 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:52732
I20260926 16:37:56.299343 18783 tablet_replica.cc:330] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: stopping tablet replica
I20260926 16:37:56.299759 18783 raft_consensus.cc:2229] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:56.300356 18783 raft_consensus.cc:2258] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:56.300994 18249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:56.301865 18249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:56.302548 18783 ts_tablet_manager.cc:1905] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:56.309526 18783 ts_tablet_manager.cc:1918] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:56.309808 18783 log.cc:1196] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/51215bb2bf44411d917e62337c24350b
I20260926 16:37:56.310490 18783 ts_tablet_manager.cc:1939] T 51215bb2bf44411d917e62337c24350b P 39b8d2513b03440f81a14bb87492fbb3: Deleting consensus metadata
I20260926 16:37:56.312491 18234 catalog_manager.cc:4841] TS 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321): tablet 51215bb2bf44411d917e62337c24350b (table test-table [id=2721bf5b0e04400c97e824896c466e2d]) successfully deleted
I20260926 16:37:56.313001 18663 tablet_service.cc:1467] Processing CreateTablet for tablet 65473ae0183b417cb275679d5d9670c0 (DEFAULT_TABLE table=test-table [id=92457524b9144bff83ef96e77777b126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:56.313093 18525 tablet_service.cc:1467] Processing CreateTablet for tablet 65473ae0183b417cb275679d5d9670c0 (DEFAULT_TABLE table=test-table [id=92457524b9144bff83ef96e77777b126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:56.313150 18385 tablet_service.cc:1467] Processing CreateTablet for tablet 65473ae0183b417cb275679d5d9670c0 (DEFAULT_TABLE table=test-table [id=92457524b9144bff83ef96e77777b126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:56.313501 18525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65473ae0183b417cb275679d5d9670c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.313546 18663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65473ae0183b417cb275679d5d9670c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.313658 18385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65473ae0183b417cb275679d5d9670c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.319628 18759 tablet_bootstrap.cc:492] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Bootstrap starting.
I20260926 16:37:56.320406 18760 tablet_bootstrap.cc:492] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Bootstrap starting.
I20260926 16:37:56.321273 18785 tablet_replica.cc:330] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: stopping tablet replica
I20260926 16:37:56.321326 18759 tablet_bootstrap.cc:654] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.321710 18785 raft_consensus.cc:2229] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:56.321990 18760 tablet_bootstrap.cc:654] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.322173 18785 raft_consensus.cc:2258] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:56.322978 18784 tablet_replica.cc:330] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: stopping tablet replica
I20260926 16:37:56.323433 18784 raft_consensus.cc:2229] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:56.324257 18784 raft_consensus.cc:2258] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:56.324314 18761 tablet_bootstrap.cc:492] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Bootstrap starting.
I20260926 16:37:56.324982 18785 ts_tablet_manager.cc:1905] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:56.325695 18761 tablet_bootstrap.cc:654] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.327908 18760 tablet_bootstrap.cc:492] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: No bootstrap required, opened a new log
I20260926 16:37:56.328064 18760 ts_tablet_manager.cc:1397] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260926 16:37:56.328866 18760 raft_consensus.cc:348] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.329077 18760 raft_consensus.cc:374] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.329159 18760 raft_consensus.cc:729] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.329427 18760 consensus_queue.cc:260] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.332456 18760 ts_tablet_manager.cc:1428] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:37:56.340337 18785 ts_tablet_manager.cc:1918] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:56.340621 18785 log.cc:1196] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/51215bb2bf44411d917e62337c24350b
I20260926 16:37:56.341120 18784 ts_tablet_manager.cc:1905] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:56.341209 18785 ts_tablet_manager.cc:1939] T 51215bb2bf44411d917e62337c24350b P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting consensus metadata
I20260926 16:37:56.343158 18235 catalog_manager.cc:4841] TS 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383): tablet 51215bb2bf44411d917e62337c24350b (table test-table [id=2721bf5b0e04400c97e824896c466e2d]) successfully deleted
I20260926 16:37:56.350926 18784 ts_tablet_manager.cc:1918] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:56.352131 18784 log.cc:1196] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/51215bb2bf44411d917e62337c24350b
I20260926 16:37:56.352623 18761 tablet_bootstrap.cc:492] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: No bootstrap required, opened a new log
I20260926 16:37:56.352787 18784 ts_tablet_manager.cc:1939] T 51215bb2bf44411d917e62337c24350b P 5a4499d7f5214491a342137c477bc0e2: Deleting consensus metadata
I20260926 16:37:56.352861 18761 ts_tablet_manager.cc:1397] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.011s
I20260926 16:37:56.353816 18761 raft_consensus.cc:348] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.354146 18761 raft_consensus.cc:374] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.354296 18761 raft_consensus.cc:729] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894b1a87ba9f4f8c92e7025a28a46b89, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.354743 18235 catalog_manager.cc:4841] TS 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163): tablet 51215bb2bf44411d917e62337c24350b (table test-table [id=2721bf5b0e04400c97e824896c466e2d]) successfully deleted
I20260926 16:37:56.354660 18761 consensus_queue.cc:260] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.356576 18759 tablet_bootstrap.cc:492] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: No bootstrap required, opened a new log
I20260926 16:37:56.356783 18759 ts_tablet_manager.cc:1397] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.006s
I20260926 16:37:56.357635 18759 raft_consensus.cc:348] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.357782 18761 ts_tablet_manager.cc:1428] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260926 16:37:56.357867 18759 raft_consensus.cc:374] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.358008 18759 raft_consensus.cc:729] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.358278 18759 consensus_queue.cc:260] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.358908 18759 ts_tablet_manager.cc:1428] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:37:56.500283 18766 raft_consensus.cc:482] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:56.500550 18766 raft_consensus.cc:504] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.502434 18766 leader_election.cc:288] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:56.510092 18758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.503664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:53713 (local address 127.10.110.3:44383)
0926 16:37:56.503870 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.503876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.504516 (+ 640us) server_negotiation.cc:407] Connection header received
0926 16:37:56.504751 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.504758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.504845 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.504942 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:56.505497 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.506273 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.507924 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.509079 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.509245 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.509318 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.509687 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.509813 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:37:56.510164 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.503452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44383 (local address 127.10.110.2:53713)
0926 16:37:56.504376 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.504394 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.504551 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.504969 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.504973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.504998 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:56.505335 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.505346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.506401 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.506405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.507760 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.507773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.509292 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.509299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.509509 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.509546 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.509649 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.509777 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":4,"thread_start_us":567,"threads_started":1}
I20260926 16:37:56.510200 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.502831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:50937 (local address 127.10.110.1:41163)
0926 16:37:56.503035 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.503041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.503385 (+ 344us) server_negotiation.cc:407] Connection header received
0926 16:37:56.503697 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.503704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.503788 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.503923 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:37:56.504675 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.505588 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.508057 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.509209 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.509342 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.509419 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.509782 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.509934 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:37:56.510870 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.502713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41163 (local address 127.10.110.2:50937)
0926 16:37:56.503290 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.503308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.503503 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.504139 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.504146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.504172 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:37:56.504501 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.504513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.505711 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.505716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.507894 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.507909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.509487 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.509491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.509613 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.509644 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.510440 (+ 796us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.510572 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":315,"threads_started":1}
I20260926 16:37:56.511123 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65473ae0183b417cb275679d5d9670c0" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" is_pre_election: true
I20260926 16:37:56.511380 18684 raft_consensus.cc:2454] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 0.
I20260926 16:37:56.511907 18478 leader_election.cc:302] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 894b1a87ba9f4f8c92e7025a28a46b89; no voters:
I20260926 16:37:56.511795 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65473ae0183b417cb275679d5d9670c0" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4499d7f5214491a342137c477bc0e2" is_pre_election: true
I20260926 16:37:56.512265 18766 raft_consensus.cc:2781] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:56.512250 18406 raft_consensus.cc:2454] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 0.
I20260926 16:37:56.512377 18766 raft_consensus.cc:482] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:56.512498 18766 raft_consensus.cc:3037] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.514813 18766 raft_consensus.cc:504] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.515375 18766 leader_election.cc:288] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:56.515771 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65473ae0183b417cb275679d5d9670c0" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4499d7f5214491a342137c477bc0e2"
I20260926 16:37:56.515933 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65473ae0183b417cb275679d5d9670c0" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
I20260926 16:37:56.516005 18406 raft_consensus.cc:3037] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.516129 18684 raft_consensus.cc:3037] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.518050 18684 raft_consensus.cc:2454] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 1.
I20260926 16:37:56.518476 18478 leader_election.cc:302] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 894b1a87ba9f4f8c92e7025a28a46b89; no voters:
I20260926 16:37:56.518819 18766 raft_consensus.cc:2781] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:56.518935 18406 raft_consensus.cc:2454] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 1.
I20260926 16:37:56.519531 18766 raft_consensus.cc:686] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 LEADER]: Becoming Leader. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Running, Role: LEADER
I20260926 16:37:56.519944 18766 consensus_queue.cc:237] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.523099 18248 catalog_manager.cc:5495] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2). New cstate: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:37:56 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440673, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.62@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:56.613277 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.594149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33365 (local address 127.0.0.1:51168)
0926 16:37:56.595516 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.595536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.595658 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.596127 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.596131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.596823 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:37:56.597047 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.597055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.598528 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.598532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.599490 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.599500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.600436 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.600440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.600567 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.604734 (+ 4167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:37:56.604757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:37:56.606851 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:37:56.610507 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:56.610518 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:56.610524 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:56.611073 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:56.611499 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:37:56.611503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:37:56.611505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:37:56.611657 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:37:56.612381 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:37:56.612388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:37:56.612544 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.612681 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.612888 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"thread_start_us":1091,"threads_started":1}
I20260926 16:37:56.613401 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.594177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.10.110.62:33365)
0926 16:37:56.595391 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.595432 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.595626 (+ 194us) server_negotiation.cc:407] Connection header received
0926 16:37:56.595877 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.595883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.595968 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.596080 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:56.597210 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.598394 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.599627 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.600277 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.600405 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.607085 (+ 6680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:56.607092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:56.607098 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:56.607134 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:56.610354 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:56.611187 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:56.611193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:56.611197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:56.611339 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:56.611781 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:56.611786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:56.611789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:56.612220 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:56.612374 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.612797 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.613022 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":779,"threads_started":1}
I20260926 16:37:56.618415 18248 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:51168:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:37:56.619438 18248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:56.626417 18525 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff5bc90bdad47acb2f1ac4f27c6f95a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c4df65a67c1431faad931d1f5588451]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:56.626420 18386 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff5bc90bdad47acb2f1ac4f27c6f95a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c4df65a67c1431faad931d1f5588451]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:56.626750 18663 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff5bc90bdad47acb2f1ac4f27c6f95a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c4df65a67c1431faad931d1f5588451]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:37:56.626981 18525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff5bc90bdad47acb2f1ac4f27c6f95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.627005 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff5bc90bdad47acb2f1ac4f27c6f95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.627308 18663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff5bc90bdad47acb2f1ac4f27c6f95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.634714 18760 tablet_bootstrap.cc:492] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: Bootstrap starting.
I20260926 16:37:56.635805 18759 tablet_bootstrap.cc:492] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: Bootstrap starting.
I20260926 16:37:56.636385 18760 tablet_bootstrap.cc:654] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.637465 18759 tablet_bootstrap.cc:654] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.638556 18761 tablet_bootstrap.cc:492] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: Bootstrap starting.
I20260926 16:37:56.640287 18761 tablet_bootstrap.cc:654] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.643666 18761 tablet_bootstrap.cc:492] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: No bootstrap required, opened a new log
I20260926 16:37:56.644104 18761 ts_tablet_manager.cc:1397] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 16:37:56.644886 18761 raft_consensus.cc:348] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.645191 18761 raft_consensus.cc:374] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.645303 18761 raft_consensus.cc:729] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894b1a87ba9f4f8c92e7025a28a46b89, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.646579 18761 consensus_queue.cc:260] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.647298 18761 ts_tablet_manager.cc:1428] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 16:37:56.647162 18767 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.647542 18767 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.648241 18760 tablet_bootstrap.cc:492] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: No bootstrap required, opened a new log
I20260926 16:37:56.648435 18760 ts_tablet_manager.cc:1397] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260926 16:37:56.649171 18760 raft_consensus.cc:348] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.649423 18760 raft_consensus.cc:374] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.649487 18760 raft_consensus.cc:729] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.649711 18760 consensus_queue.cc:260] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.650110 18766 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.650404 18766 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.652320 18760 ts_tablet_manager.cc:1428] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:37:56.653322 18759 tablet_bootstrap.cc:492] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: No bootstrap required, opened a new log
I20260926 16:37:56.653503 18759 ts_tablet_manager.cc:1397] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.006s
I20260926 16:37:56.654306 18759 raft_consensus.cc:348] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.654531 18759 raft_consensus.cc:374] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.654610 18759 raft_consensus.cc:729] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.654853 18759 consensus_queue.cc:260] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.655262 18770 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.655474 18759 ts_tablet_manager.cc:1428] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 16:37:56.655637 18770 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.674631 18766 raft_consensus.cc:482] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:56.674860 18766 raft_consensus.cc:504] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.675451 18766 leader_election.cc:288] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:56.676084 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff5bc90bdad47acb2f1ac4f27c6f95a" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" is_pre_election: true
I20260926 16:37:56.676142 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff5bc90bdad47acb2f1ac4f27c6f95a" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4499d7f5214491a342137c477bc0e2" is_pre_election: true
I20260926 16:37:56.676386 18684 raft_consensus.cc:2454] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 0.
I20260926 16:37:56.676393 18406 raft_consensus.cc:2454] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 0.
I20260926 16:37:56.676777 18478 leader_election.cc:302] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 894b1a87ba9f4f8c92e7025a28a46b89; no voters:
I20260926 16:37:56.677080 18766 raft_consensus.cc:2781] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:56.677193 18766 raft_consensus.cc:482] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:56.677300 18766 raft_consensus.cc:3037] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.680370 18766 raft_consensus.cc:504] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.681069 18766 leader_election.cc:288] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:56.681582 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff5bc90bdad47acb2f1ac4f27c6f95a" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4499d7f5214491a342137c477bc0e2"
I20260926 16:37:56.681641 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff5bc90bdad47acb2f1ac4f27c6f95a" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
I20260926 16:37:56.681846 18406 raft_consensus.cc:3037] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.681886 18684 raft_consensus.cc:3037] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:56.684702 18684 raft_consensus.cc:2454] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 1.
I20260926 16:37:56.684969 18406 raft_consensus.cc:2454] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 1.
I20260926 16:37:56.685141 18478 leader_election.cc:302] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 894b1a87ba9f4f8c92e7025a28a46b89; no voters:
I20260926 16:37:56.685432 18766 raft_consensus.cc:2781] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:56.685600 18766 raft_consensus.cc:686] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [term 1 LEADER]: Becoming Leader. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Running, Role: LEADER
I20260926 16:37:56.685853 18766 consensus_queue.cc:237] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.687592 18791 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: TxnStatusTablet state changed. Reason: New leader 39b8d2513b03440f81a14bb87492fbb3. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.687996 18791 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:56.688634 18804 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:37:56.688822 18804 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:37:56.689150 18249 catalog_manager.cc:5495] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2). New cstate: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:56.708271 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.702869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:36321 (local address 127.0.0.1:53268)
0926 16:37:56.703065 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.703084 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.703190 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.703730 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.703735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.703748 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:56.703978 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.703990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.705014 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.705019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.705991 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.706002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.706690 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.706696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.706866 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.706919 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:56.707713 (+ 794us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:56.707733 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.707819 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.707989 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 16:37:56.708313 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.702967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.10.110.2:36321)
0926 16:37:56.703413 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.703417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.703432 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:37:56.703501 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.703504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.703574 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.703686 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:56.704110 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.704883 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.706122 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.706542 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.706669 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.707099 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:56.707608 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:56.707734 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.707873 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.707952 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342}
I20260926 16:37:56.712467 18684 raft_consensus.cc:1261] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Refusing update from remote peer 39b8d2513b03440f81a14bb87492fbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:56.712781 18406 raft_consensus.cc:1261] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Refusing update from remote peer 39b8d2513b03440f81a14bb87492fbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:56.713261 18766 consensus_queue.cc:1035] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:56.713781 18791 consensus_queue.cc:1035] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:56.725488 18770 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: TxnStatusTablet state changed. Reason: New leader 39b8d2513b03440f81a14bb87492fbb3. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.725800 18770 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.731220 18766 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.731534 18766 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:56.731714 18774 mvcc.cc:205] Tried to move back new op lower bound from 7333645011806248960 to 7333645011707793408. Current Snapshot: MvccSnapshot[applied={T|T < 7333645011806248960 or (T in {7333645011806248960})}]
I20260926 16:37:56.732898 18770 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.733155 18770 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 5a4499d7f5214491a342137c477bc0e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.744146 18767 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: TxnStatusTablet state changed. Reason: New leader 39b8d2513b03440f81a14bb87492fbb3. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.744445 18767 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.747644 18791 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.747933 18791 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 39b8d2513b03440f81a14bb87492fbb3: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:37:56.748612 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.742292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.10.110.2:36321)
0926 16:37:56.742540 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.742545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.742562 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:37:56.742649 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.742653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.742732 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.742821 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:56.743396 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.744565 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.746360 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.746957 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.747063 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.747611 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:56.747739 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:56.747848 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.748266 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.748338 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 16:37:56.756455 18767 tablet_replica.cc:439] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } } }
I20260926 16:37:56.756707 18767 tablet_replica.cc:442] T 6ff5bc90bdad47acb2f1ac4f27c6f95a P 894b1a87ba9f4f8c92e7025a28a46b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:37:56.767462 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.762108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:40689 (local address 127.10.110.2:36321)
0926 16:37:56.762359 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.762367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.762681 (+ 314us) server_negotiation.cc:407] Connection header received
0926 16:37:56.762848 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.762853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.762912 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.762977 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:56.763490 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.764397 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.765773 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.766267 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.766385 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.766777 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:56.766900 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:56.766969 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.767082 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.767143 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":88}
I20260926 16:37:56.767896 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.762050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:36321 (local address 127.10.110.2:40689)
0926 16:37:56.762594 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 16:37:56.762608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:37:56.762733 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:37:56.762995 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:37:56.762999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:37:56.763009 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:37:56.763364 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.763373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.764508 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.764513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.765649 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:37:56.765661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.766446 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.766452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:37:56.766574 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.766626 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:37:56.766985 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:37:56.766994 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:37:56.767058 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 16:37:56.767177 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":20,"thread_start_us":319,"threads_started":1}
I20260926 16:37:56.778509 18684 raft_consensus.cc:1261] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Refusing update from remote peer 39b8d2513b03440f81a14bb87492fbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:56.778682 18406 raft_consensus.cc:1261] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Refusing update from remote peer 39b8d2513b03440f81a14bb87492fbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:56.779233 18766 consensus_queue.cc:1035] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:56.779527 18791 consensus_queue.cc:1035] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:56.839022 18249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:56.839237 18249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:56.842010 18249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Sending DeleteTablet for 3 replicas of tablet 65473ae0183b417cb275679d5d9670c0
I20260926 16:37:56.842839 18386 tablet_service.cc:1514] Processing DeleteTablet for tablet 65473ae0183b417cb275679d5d9670c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:56 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 16:37:56.842856 18524 tablet_service.cc:1514] Processing DeleteTablet for tablet 65473ae0183b417cb275679d5d9670c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:56 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53256
I20260926 16:37:56.843122 18784 tablet_replica.cc:330] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: stopping tablet replica
I20260926 16:37:56.843226 18663 tablet_service.cc:1514] Processing DeleteTablet for tablet 65473ae0183b417cb275679d5d9670c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:56 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:52732
I20260926 16:37:56.843307 18784 raft_consensus.cc:2229] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:56.843484 18784 raft_consensus.cc:2258] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:56.843740 18785 tablet_replica.cc:330] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: stopping tablet replica
I20260926 16:37:56.843954 18785 raft_consensus.cc:2229] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:56.844146 18785 raft_consensus.cc:2258] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:56.845468 18784 ts_tablet_manager.cc:1905] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:56.849077 18785 ts_tablet_manager.cc:1905] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:56.852380 18784 ts_tablet_manager.cc:1918] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:56.852535 18784 log.cc:1196] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/65473ae0183b417cb275679d5d9670c0
I20260926 16:37:56.852927 18784 ts_tablet_manager.cc:1939] T 65473ae0183b417cb275679d5d9670c0 P 5a4499d7f5214491a342137c477bc0e2: Deleting consensus metadata
I20260926 16:37:56.854365 18235 catalog_manager.cc:4841] TS 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163): tablet 65473ae0183b417cb275679d5d9670c0 (table test-table [id=92457524b9144bff83ef96e77777b126]) successfully deleted
I20260926 16:37:56.855686 18785 ts_tablet_manager.cc:1918] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:56.855852 18785 log.cc:1196] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/65473ae0183b417cb275679d5d9670c0
I20260926 16:37:56.856256 18785 ts_tablet_manager.cc:1939] T 65473ae0183b417cb275679d5d9670c0 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting consensus metadata
I20260926 16:37:56.857226 18235 catalog_manager.cc:4841] TS 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383): tablet 65473ae0183b417cb275679d5d9670c0 (table test-table [id=92457524b9144bff83ef96e77777b126]) successfully deleted
I20260926 16:37:56.860265 18816 tablet_replica.cc:330] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: stopping tablet replica
I20260926 16:37:56.860466 18816 raft_consensus.cc:2229] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:56.861049 18816 raft_consensus.cc:2258] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:56.862027 18816 ts_tablet_manager.cc:1905] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:56.869562 18816 ts_tablet_manager.cc:1918] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:56.869706 18816 log.cc:1196] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/65473ae0183b417cb275679d5d9670c0
I20260926 16:37:56.870199 18816 ts_tablet_manager.cc:1939] T 65473ae0183b417cb275679d5d9670c0 P 39b8d2513b03440f81a14bb87492fbb3: Deleting consensus metadata
I20260926 16:37:56.871397 18234 catalog_manager.cc:4841] TS 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321): tablet 65473ae0183b417cb275679d5d9670c0 (table test-table [id=92457524b9144bff83ef96e77777b126]) successfully deleted
Sep 26 16:37:56 dist-test-slave-whjp krb5kdc[18204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:37:56 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:37:56 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:37:56.904155 18798 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:37:56.891630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.10.110.62:33365)
0926 16:37:56.891842 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.891850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.892258 (+ 408us) server_negotiation.cc:407] Connection header received
0926 16:37:56.892410 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.892414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.892489 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.892573 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:56.893850 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.894972 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.896797 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.897455 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.897582 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.903921 (+ 6339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51174: BlockingRecv error: recv got EOF from 127.0.0.1:51174 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 16:37:56 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM
I20260926 16:37:56.936866 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:56.913143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.10.110.62:33365)
0926 16:37:56.913309 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:56.913315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:56.920309 (+ 6994us) server_negotiation.cc:407] Connection header received
0926 16:37:56.920470 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:56.920476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:56.920557 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:56.920663 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:56.922213 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.923340 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.925208 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:56.925817 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:56.925939 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:56.930802 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:56.930810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:56.930815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:56.930859 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:56.934116 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:56.934868 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:56.934874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:56.934878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:56.935018 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:56.935364 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:56.935369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:56.935372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:56.935723 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:56.935990 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:56.936257 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:37:56.936510 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:37:56.940696 18249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:56.941409 18249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:56.948657 18386 tablet_service.cc:1467] Processing CreateTablet for tablet 706529720d3845c9a8c149d30d6ad4b1 (DEFAULT_TABLE table=test-table [id=6d523123db164f29bc829baf762fa0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:56.949110 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706529720d3845c9a8c149d30d6ad4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.953858 18524 tablet_service.cc:1467] Processing CreateTablet for tablet 706529720d3845c9a8c149d30d6ad4b1 (DEFAULT_TABLE table=test-table [id=6d523123db164f29bc829baf762fa0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:56.954002 18663 tablet_service.cc:1467] Processing CreateTablet for tablet 706529720d3845c9a8c149d30d6ad4b1 (DEFAULT_TABLE table=test-table [id=6d523123db164f29bc829baf762fa0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:56.954353 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706529720d3845c9a8c149d30d6ad4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.954475 18663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706529720d3845c9a8c149d30d6ad4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:56.959712 18760 tablet_bootstrap.cc:492] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Bootstrap starting.
I20260926 16:37:56.961694 18760 tablet_bootstrap.cc:654] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.963724 18761 tablet_bootstrap.cc:492] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Bootstrap starting.
I20260926 16:37:56.963753 18759 tablet_bootstrap.cc:492] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Bootstrap starting.
I20260926 16:37:56.965184 18761 tablet_bootstrap.cc:654] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.965301 18759 tablet_bootstrap.cc:654] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:56.965533 18760 tablet_bootstrap.cc:492] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: No bootstrap required, opened a new log
I20260926 16:37:56.965737 18760 ts_tablet_manager.cc:1397] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:37:56.966638 18760 raft_consensus.cc:348] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.967056 18760 raft_consensus.cc:374] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.967659 18760 raft_consensus.cc:729] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.968104 18760 consensus_queue.cc:260] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.968856 18760 ts_tablet_manager.cc:1428] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:56.971006 18761 tablet_bootstrap.cc:492] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: No bootstrap required, opened a new log
I20260926 16:37:56.971168 18761 ts_tablet_manager.cc:1397] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 16:37:56.971984 18761 raft_consensus.cc:348] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.972242 18761 raft_consensus.cc:374] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.972332 18761 raft_consensus.cc:729] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894b1a87ba9f4f8c92e7025a28a46b89, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.972589 18761 consensus_queue.cc:260] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.973104 18761 ts_tablet_manager.cc:1428] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:37:56.977042 18759 tablet_bootstrap.cc:492] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: No bootstrap required, opened a new log
I20260926 16:37:56.977196 18759 ts_tablet_manager.cc:1397] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260926 16:37:56.977850 18759 raft_consensus.cc:348] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.978046 18759 raft_consensus.cc:374] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:56.978133 18759 raft_consensus.cc:729] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Initialized, Role: FOLLOWER
I20260926 16:37:56.978346 18759 consensus_queue.cc:260] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:56.978788 18759 ts_tablet_manager.cc:1428] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 16:37:57.117919 18770 raft_consensus.cc:482] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:57.118131 18770 raft_consensus.cc:504] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.118798 18770 leader_election.cc:288] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:57.119447 18545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706529720d3845c9a8c149d30d6ad4b1" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b8d2513b03440f81a14bb87492fbb3" is_pre_election: true
I20260926 16:37:57.119529 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706529720d3845c9a8c149d30d6ad4b1" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" is_pre_election: true
I20260926 16:37:57.119715 18545 raft_consensus.cc:2454] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 0.
I20260926 16:37:57.119760 18684 raft_consensus.cc:2454] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 0.
I20260926 16:37:57.120225 18337 leader_election.cc:302] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 5a4499d7f5214491a342137c477bc0e2; no voters:
I20260926 16:37:57.120512 18770 raft_consensus.cc:2781] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:57.120626 18770 raft_consensus.cc:482] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:57.120745 18770 raft_consensus.cc:3037] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:57.122792 18770 raft_consensus.cc:504] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.123418 18770 leader_election.cc:288] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 election: Requested vote from peers 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:57.123943 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706529720d3845c9a8c149d30d6ad4b1" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
I20260926 16:37:57.123977 18545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706529720d3845c9a8c149d30d6ad4b1" candidate_uuid: "5a4499d7f5214491a342137c477bc0e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b8d2513b03440f81a14bb87492fbb3"
I20260926 16:37:57.124176 18684 raft_consensus.cc:3037] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:57.124186 18545 raft_consensus.cc:3037] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:57.126576 18545 raft_consensus.cc:2454] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 1.
I20260926 16:37:57.126995 18684 raft_consensus.cc:2454] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4499d7f5214491a342137c477bc0e2 in term 1.
I20260926 16:37:57.127152 18337 leader_election.cc:302] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 5a4499d7f5214491a342137c477bc0e2; no voters:
I20260926 16:37:57.127498 18770 raft_consensus.cc:2781] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:57.127974 18770 raft_consensus.cc:686] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Running, Role: LEADER
I20260926 16:37:57.128260 18770 consensus_queue.cc:237] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.131070 18248 catalog_manager.cc:5495] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "5a4499d7f5214491a342137c477bc0e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:57.153317 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:57.147433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.10.110.1:41163)
0926 16:37:57.148081 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:57.148088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:57.148113 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:37:57.148183 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:57.148188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:57.148272 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:57.148390 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:57.148668 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.149791 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.151138 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.151773 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.151943 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:57.152471 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:57.152618 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:57.152709 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:57.152938 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 16:37:57.153017 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":338,"threads_started":1}
I20260926 16:37:57.156001 18684 raft_consensus.cc:1261] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Refusing update from remote peer 5a4499d7f5214491a342137c477bc0e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:57.155993 18545 raft_consensus.cc:1261] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Refusing update from remote peer 5a4499d7f5214491a342137c477bc0e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:57.156554 18770 consensus_queue.cc:1035] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:57.156845 18832 consensus_queue.cc:1035] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:57.168848 18249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:57.169204 18249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:57.175001 18249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Sending DeleteTablet for 3 replicas of tablet 706529720d3845c9a8c149d30d6ad4b1
I20260926 16:37:57.175748 18386 tablet_service.cc:1514] Processing DeleteTablet for tablet 706529720d3845c9a8c149d30d6ad4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:57 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 16:37:57.176064 18784 tablet_replica.cc:330] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: stopping tablet replica
I20260926 16:37:57.176115 18663 tablet_service.cc:1514] Processing DeleteTablet for tablet 706529720d3845c9a8c149d30d6ad4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:57 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:52732
I20260926 16:37:57.176234 18784 raft_consensus.cc:2229] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:57.176359 18785 tablet_replica.cc:330] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: stopping tablet replica
I20260926 16:37:57.176518 18785 raft_consensus.cc:2229] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:57.176523 18784 raft_consensus.cc:2258] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:57.176730 18785 raft_consensus.cc:2258] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:57.176956 18524 tablet_service.cc:1514] Processing DeleteTablet for tablet 706529720d3845c9a8c149d30d6ad4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:57 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53256
I20260926 16:37:57.177232 18816 tablet_replica.cc:330] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: stopping tablet replica
I20260926 16:37:57.177585 18784 ts_tablet_manager.cc:1905] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:57.178000 18785 ts_tablet_manager.cc:1905] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:57.177999 18249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:37:57.178632 18249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:37:57.181109 18816 raft_consensus.cc:2229] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:57.181596 18816 pending_rounds.cc:62] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Trying to abort 2 pending ops.
I20260926 16:37:57.181741 18816 pending_rounds.cc:68] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333645013625061376 op_type: WRITE_OP write_request { tablet_id: "706529720d3845c9a8c149d30d6ad4b1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\221\354\337\325\006\"8\n\ntest-admin\022*\n 6d523123db164f29bc829baf762fa0fc\020\001\030\001 \001(\001" signature: "<redacted>""Q\303R\213\304\326m&0y\307\264,\373K\177\211\335\ne\214\000\020\333\226cQ\212 \033\344\246OG\243\362-\237m\372\255\224\250\337oR\306M\322\305\262\206\264\273T\226\214<\010\367`\212\232\016" signing_key_seq_num: 0 } } request_id { client_id: "4037503fd95944a5859389019ed884af" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 16:37:57.182452 18816 pending_rounds.cc:68] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333645013519892480 op_type: NO_OP noop_request { }
I20260926 16:37:57.182607 18816 raft_consensus.cc:2866] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 16:37:57.182736 18816 raft_consensus.cc:2258] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:57.183980 18784 ts_tablet_manager.cc:1918] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:57.184108 18784 log.cc:1196] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/706529720d3845c9a8c149d30d6ad4b1
I20260926 16:37:57.184476 18784 ts_tablet_manager.cc:1939] T 706529720d3845c9a8c149d30d6ad4b1 P 5a4499d7f5214491a342137c477bc0e2: Deleting consensus metadata
I20260926 16:37:57.185626 18235 catalog_manager.cc:4841] TS 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163): tablet 706529720d3845c9a8c149d30d6ad4b1 (table test-table [id=6d523123db164f29bc829baf762fa0fc]) successfully deleted
I20260926 16:37:57.187740 18785 ts_tablet_manager.cc:1918] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:57.187860 18785 log.cc:1196] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/706529720d3845c9a8c149d30d6ad4b1
I20260926 16:37:57.188582 18816 ts_tablet_manager.cc:1905] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:57.189956 18524 tablet_service.cc:1467] Processing CreateTablet for tablet cec94e0933274d4ea63afa36e5aba6b9 (DEFAULT_TABLE table=test-table [id=96029fbf066346ec892bd89939e537c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:57.190366 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec94e0933274d4ea63afa36e5aba6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:57.192737 18663 tablet_service.cc:1467] Processing CreateTablet for tablet cec94e0933274d4ea63afa36e5aba6b9 (DEFAULT_TABLE table=test-table [id=96029fbf066346ec892bd89939e537c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:57.193112 18663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec94e0933274d4ea63afa36e5aba6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:57.193694 18816 ts_tablet_manager.cc:1918] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:37:57.193841 18816 log.cc:1196] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/706529720d3845c9a8c149d30d6ad4b1
I20260926 16:37:57.194567 18785 ts_tablet_manager.cc:1939] T 706529720d3845c9a8c149d30d6ad4b1 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting consensus metadata
I20260926 16:37:57.195611 18235 catalog_manager.cc:4841] TS 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383): tablet 706529720d3845c9a8c149d30d6ad4b1 (table test-table [id=6d523123db164f29bc829baf762fa0fc]) successfully deleted
I20260926 16:37:57.194258 18816 ts_tablet_manager.cc:1939] T 706529720d3845c9a8c149d30d6ad4b1 P 39b8d2513b03440f81a14bb87492fbb3: Deleting consensus metadata
I20260926 16:37:57.196897 18234 catalog_manager.cc:4841] TS 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321): tablet 706529720d3845c9a8c149d30d6ad4b1 (table test-table [id=6d523123db164f29bc829baf762fa0fc]) successfully deleted
I20260926 16:37:57.197180 18760 tablet_bootstrap.cc:492] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Bootstrap starting.
I20260926 16:37:57.198297 18761 tablet_bootstrap.cc:492] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Bootstrap starting.
I20260926 16:37:57.198812 18760 tablet_bootstrap.cc:654] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:57.199033 18386 tablet_service.cc:1467] Processing CreateTablet for tablet cec94e0933274d4ea63afa36e5aba6b9 (DEFAULT_TABLE table=test-table [id=96029fbf066346ec892bd89939e537c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:37:57.199429 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec94e0933274d4ea63afa36e5aba6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:37:57.199738 18761 tablet_bootstrap.cc:654] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:57.207026 18759 tablet_bootstrap.cc:492] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Bootstrap starting.
I20260926 16:37:57.208715 18759 tablet_bootstrap.cc:654] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Neither blocks nor log segments found. Creating new log.
I20260926 16:37:57.211723 18761 tablet_bootstrap.cc:492] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: No bootstrap required, opened a new log
I20260926 16:37:57.211946 18761 ts_tablet_manager.cc:1397] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260926 16:37:57.212721 18761 raft_consensus.cc:348] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.212970 18761 raft_consensus.cc:374] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:57.213043 18761 raft_consensus.cc:729] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894b1a87ba9f4f8c92e7025a28a46b89, State: Initialized, Role: FOLLOWER
I20260926 16:37:57.213300 18761 consensus_queue.cc:260] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.214917 18761 ts_tablet_manager.cc:1428] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:37:57.220957 18759 tablet_bootstrap.cc:492] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: No bootstrap required, opened a new log
I20260926 16:37:57.221115 18760 tablet_bootstrap.cc:492] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: No bootstrap required, opened a new log
I20260926 16:37:57.221127 18759 ts_tablet_manager.cc:1397] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260926 16:37:57.221272 18760 ts_tablet_manager.cc:1397] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.009s
I20260926 16:37:57.221876 18760 raft_consensus.cc:348] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.221870 18759 raft_consensus.cc:348] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.222085 18760 raft_consensus.cc:374] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:57.222100 18759 raft_consensus.cc:374] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:37:57.222160 18760 raft_consensus.cc:729] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Initialized, Role: FOLLOWER
I20260926 16:37:57.222162 18759 raft_consensus.cc:729] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4499d7f5214491a342137c477bc0e2, State: Initialized, Role: FOLLOWER
I20260926 16:37:57.222410 18760 consensus_queue.cc:260] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.222417 18759 consensus_queue.cc:260] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.222839 18760 ts_tablet_manager.cc:1428] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 16:37:57.225863 18759 ts_tablet_manager.cc:1428] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260926 16:37:57.375031 18814 raft_consensus.cc:482] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:37:57.375279 18814 raft_consensus.cc:504] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.376129 18814 leader_election.cc:288] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:57.376803 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec94e0933274d4ea63afa36e5aba6b9" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4499d7f5214491a342137c477bc0e2" is_pre_election: true
I20260926 16:37:57.376916 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec94e0933274d4ea63afa36e5aba6b9" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" is_pre_election: true
I20260926 16:37:57.377050 18406 raft_consensus.cc:2454] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 0.
I20260926 16:37:57.377156 18684 raft_consensus.cc:2454] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 0.
I20260926 16:37:57.377553 18478 leader_election.cc:302] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 5a4499d7f5214491a342137c477bc0e2; no voters:
I20260926 16:37:57.377879 18814 raft_consensus.cc:2781] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:37:57.378006 18814 raft_consensus.cc:482] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:37:57.378095 18814 raft_consensus.cc:3037] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:57.381259 18814 raft_consensus.cc:504] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.381965 18814 leader_election.cc:288] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163), 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383)
I20260926 16:37:57.382512 18684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec94e0933274d4ea63afa36e5aba6b9" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "894b1a87ba9f4f8c92e7025a28a46b89"
I20260926 16:37:57.382692 18684 raft_consensus.cc:3037] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:57.384105 18406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec94e0933274d4ea63afa36e5aba6b9" candidate_uuid: "39b8d2513b03440f81a14bb87492fbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4499d7f5214491a342137c477bc0e2"
I20260926 16:37:57.384322 18406 raft_consensus.cc:3037] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:37:57.384912 18684 raft_consensus.cc:2454] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 1.
I20260926 16:37:57.385315 18478 leader_election.cc:302] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b8d2513b03440f81a14bb87492fbb3, 894b1a87ba9f4f8c92e7025a28a46b89; no voters:
I20260926 16:37:57.385560 18814 raft_consensus.cc:2781] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:37:57.385982 18814 raft_consensus.cc:686] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 LEADER]: Becoming Leader. State: Replica: 39b8d2513b03440f81a14bb87492fbb3, State: Running, Role: LEADER
I20260926 16:37:57.386459 18814 consensus_queue.cc:237] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } }
I20260926 16:37:57.387096 18406 raft_consensus.cc:2454] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b8d2513b03440f81a14bb87492fbb3 in term 1.
I20260926 16:37:57.389000 18249 catalog_manager.cc:5495] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2). New cstate: current_term: 1 leader_uuid: "39b8d2513b03440f81a14bb87492fbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39b8d2513b03440f81a14bb87492fbb3" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 } health_report { overall_health: UNKNOWN } } }
I20260926 16:37:57.421490 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:57.414397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.10.110.2:36321)
0926 16:37:57.415096 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:57.415103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:57.415130 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:37:57.415204 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:57.415209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:57.415288 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:57.415396 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:37:57.415895 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.417110 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.418999 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.419731 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.419881 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:57.420432 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:37:57.420564 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:37:57.420648 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:57.420823 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 16:37:57.420910 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":354,"threads_started":1}
I20260926 16:37:57.437852 18684 raft_consensus.cc:1261] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Refusing update from remote peer 39b8d2513b03440f81a14bb87492fbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:57.437975 18406 raft_consensus.cc:1261] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Refusing update from remote peer 39b8d2513b03440f81a14bb87492fbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:37:57.438469 18845 consensus_queue.cc:1035] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "894b1a87ba9f4f8c92e7025a28a46b89" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:57.438844 18814 consensus_queue.cc:1035] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4499d7f5214491a342137c477bc0e2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:37:57.498461 18249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:57.498693 18249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:37:57.501616 18249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0c7df1aef4a4add8e05a89ae823a931: Sending DeleteTablet for 3 replicas of tablet cec94e0933274d4ea63afa36e5aba6b9
I20260926 16:37:57.502424 18524 tablet_service.cc:1514] Processing DeleteTablet for tablet cec94e0933274d4ea63afa36e5aba6b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:57 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:53256
I20260926 16:37:57.502535 18386 tablet_service.cc:1514] Processing DeleteTablet for tablet cec94e0933274d4ea63afa36e5aba6b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:57 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:43252
I20260926 16:37:57.502563 18663 tablet_service.cc:1514] Processing DeleteTablet for tablet cec94e0933274d4ea63afa36e5aba6b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:37:57 UTC) from {username='oryx', principal='oryx/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:52732
I20260926 16:37:57.502722 18816 tablet_replica.cc:330] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: stopping tablet replica
I20260926 16:37:57.502761 18784 tablet_replica.cc:330] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: stopping tablet replica
I20260926 16:37:57.502868 18785 tablet_replica.cc:330] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: stopping tablet replica
I20260926 16:37:57.502902 18816 raft_consensus.cc:2229] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:37:57.502944 18784 raft_consensus.cc:2229] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:57.503053 18785 raft_consensus.cc:2229] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:37:57.503103 18784 raft_consensus.cc:2258] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:57.503219 18816 raft_consensus.cc:2258] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:57.503232 18785 raft_consensus.cc:2258] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:37:57.504426 18816 ts_tablet_manager.cc:1905] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:57.504427 18785 ts_tablet_manager.cc:1905] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:57.509562 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 18320
I20260926 16:37:57.511039 18785 ts_tablet_manager.cc:1918] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:57.511152 18816 ts_tablet_manager.cc:1918] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:57.511195 18785 log.cc:1196] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/cec94e0933274d4ea63afa36e5aba6b9
I20260926 16:37:57.511281 18816 log.cc:1196] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/cec94e0933274d4ea63afa36e5aba6b9
I20260926 16:37:57.511577 18785 ts_tablet_manager.cc:1939] T cec94e0933274d4ea63afa36e5aba6b9 P 894b1a87ba9f4f8c92e7025a28a46b89: Deleting consensus metadata
I20260926 16:37:57.511662 18816 ts_tablet_manager.cc:1939] T cec94e0933274d4ea63afa36e5aba6b9 P 39b8d2513b03440f81a14bb87492fbb3: Deleting consensus metadata
I20260926 16:37:57.512784 18234 catalog_manager.cc:4841] TS 39b8d2513b03440f81a14bb87492fbb3 (127.10.110.2:36321): tablet cec94e0933274d4ea63afa36e5aba6b9 (table test-table [id=96029fbf066346ec892bd89939e537c5]) successfully deleted
I20260926 16:37:57.512961 18235 catalog_manager.cc:4841] TS 894b1a87ba9f4f8c92e7025a28a46b89 (127.10.110.3:44383): tablet cec94e0933274d4ea63afa36e5aba6b9 (table test-table [id=96029fbf066346ec892bd89939e537c5]) successfully deleted
I20260926 16:37:57.514520 18784 ts_tablet_manager.cc:1905] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:37:57.520604 18784 ts_tablet_manager.cc:1918] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:37:57.520777 18784 log.cc:1196] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/cec94e0933274d4ea63afa36e5aba6b9
I20260926 16:37:57.521154 18784 ts_tablet_manager.cc:1939] T cec94e0933274d4ea63afa36e5aba6b9 P 5a4499d7f5214491a342137c477bc0e2: Deleting consensus metadata
Sep 26 16:37:57 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.1@KRBTEST.COM
I20260926 16:37:57.522959 18235 catalog_manager.cc:4841] TS 5a4499d7f5214491a342137c477bc0e2 (127.10.110.1:41163): tablet cec94e0933274d4ea63afa36e5aba6b9 (table test-table [id=96029fbf066346ec892bd89939e537c5]) successfully deleted
I20260926 16:37:57.529634 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:57.510819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.10.110.1:41163)
0926 16:37:57.510997 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:57.511004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:57.512980 (+ 1976us) server_negotiation.cc:407] Connection header received
0926 16:37:57.513254 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:57.513258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:57.513332 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:57.513427 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:57.514901 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.516146 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.517860 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.518500 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.518600 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:57.523977 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:57.523992 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:57.524008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:57.524047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:57.527015 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:57.527646 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:57.527653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:57.527656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:57.527777 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:57.528184 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:57.528188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:57.528190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:57.528549 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:57.528821 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:57.529114 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:37:57.529344 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:37:57.530201 18451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:57.650998 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 18320
I20260926 16:37:57.669242 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 18460
Sep 26 16:37:57 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.2@KRBTEST.COM
I20260926 16:37:57.684019 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:57.669968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.10.110.2:36321)
0926 16:37:57.670140 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:57.670145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:57.670238 (+ 93us) server_negotiation.cc:407] Connection header received
0926 16:37:57.670408 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:57.670412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:57.670470 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:57.670549 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:57.671923 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.672629 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.673806 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.674249 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.674373 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:57.678295 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:57.678309 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:57.678325 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:57.678361 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:57.681035 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:57.681691 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:57.681699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:57.681701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:57.681847 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:57.682238 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:57.682243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:57.682246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:57.682680 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:57.682959 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:57.683235 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 16:37:57.683499 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:37:57.684649 18590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:57.823876 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 18460
I20260926 16:37:57.844527 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 18599
Sep 26 16:37:57 dist-test-slave-whjp krb5kdc[18204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.110.3@KRBTEST.COM
I20260926 16:37:57.858877 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:37:57.845284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.10.110.3:44383)
0926 16:37:57.845689 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 16:37:57.845693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:37:57.845709 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:37:57.845760 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:37:57.845764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:37:57.845825 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:37:57.845937 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:37:57.847092 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.847780 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.848982 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:37:57.849374 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:37:57.849460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:37:57.853566 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:37:57.853581 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:37:57.853597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:37:57.853631 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:37:57.856410 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:57.857104 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:57.857110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:57.857112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:57.857228 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:37:57.857565 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:37:57.857568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:37:57.857570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:37:57.857892 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:37:57.858102 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 16:37:57.858413 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 16:37:57.858566 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":208,"threads_started":1}
I20260926 16:37:57.859512 18729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:57.953934 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 18599
I20260926 16:37:57.968014 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 18217
I20260926 16:37:57.969161 18289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:37:58.084004 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 18217
2026-09-26T16:37:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6409 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:58 dist-test-slave-whjp krb5kdc[18860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:58 dist-test-slave-whjp krb5kdc[18860](info): set up 2 sockets
Sep 26 16:37:58 dist-test-slave-whjp krb5kdc[18860](info): commencing operation
krb5kdc: starting...
W20260926 16:37:59.097234 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:37:59 dist-test-slave-whjp krb5kdc[18860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:37:59.299719 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38371
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:38371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:37:59.621131 18873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:37:59.621434 18873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:37:59.621562 18873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:37:59.628623 18873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:37:59.628736 18873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:37:59.628810 18873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:37:59.628883 18873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:37:59.628952 18873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:37:59.638726 18873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:38371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:37:59.640012 18873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:37:59.641278 18873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:37:59.652513 18881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:59.652580 18878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:37:59.652560 18879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:37:59.653499 18873 server_base.cc:1034] running on GCE node
I20260926 16:37:59.654274 18873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:37:59.655648 18873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:37:59.656847 18873 hybrid_clock.cc:648] HybridClock initialized: now 1790440679656797 us; error 46 us; skew 500 ppm
I20260926 16:37:59.657179 18873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T16:37:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1645 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:37:59 dist-test-slave-whjp krb5kdc[18887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:37:59 dist-test-slave-whjp krb5kdc[18887](info): set up 2 sockets
Sep 26 16:37:59 dist-test-slave-whjp krb5kdc[18887](info): commencing operation
krb5kdc: starting...
W20260926 16:38:00.729246 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:38:00 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:38:00.931191 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33509
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:01.255142 18900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:01.255448 18900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:01.255540 18900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:01.262789 18900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:38:01.262907 18900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:01.262959 18900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:01.263015 18900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:38:01.263055 18900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:38:01.272940 18900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:01.274215 18900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:01.275442 18900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:01.286846 18908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:01.286904 18905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:01.286933 18906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:01.287942 18900 server_base.cc:1034] running on GCE node
I20260926 16:38:01.288729 18900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:01.290218 18900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:01.291424 18900 hybrid_clock.cc:648] HybridClock initialized: now 1790440681291358 us; error 59 us; skew 500 ppm
I20260926 16:38:01.291764 18900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:01 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440681, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:01.294736 18900 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:38:01.296176 18900 webserver.cc:460] Webserver started at http://127.10.110.62:38367/ using document root <none> and password file <none>
I20260926 16:38:01.296832 18900 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:01.296937 18900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:01.297195 18900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:01.300171 18900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "6cc399f75b4584de5f96112c51f53b41"
server_key_iv: "6eef72c80ec521a27e1356c4d05e57c1"
server_key_version: "encryptionkey@0"
I20260926 16:38:01.300917 18900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "6cc399f75b4584de5f96112c51f53b41"
server_key_iv: "6eef72c80ec521a27e1356c4d05e57c1"
server_key_version: "encryptionkey@0"
I20260926 16:38:01.306087 18900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:38:01.309455 18915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:01.310756 18900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:38:01.310925 18900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "6cc399f75b4584de5f96112c51f53b41"
server_key_iv: "6eef72c80ec521a27e1356c4d05e57c1"
server_key_version: "encryptionkey@0"
I20260926 16:38:01.311102 18900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:01.329015 18900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:01.333014 18900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:01.333276 18900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:01.350057 18900 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33509
I20260926 16:38:01.350064 18966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33509 every 8 connection(s)
I20260926 16:38:01.351368 18900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:38:01.351744 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 18900
I20260926 16:38:01.351976 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:38:01.352775 10680 external_mini_cluster.cc:1428] Setting key 46e9b3dd716faef475bc3b067bdf116b
I20260926 16:38:01.355752 18967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:38:01 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:01.365742 18967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Bootstrap starting.
I20260926 16:38:01.368924 18967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:01.370002 18967 log.cc:824] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:01.373154 18967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: No bootstrap required, opened a new log
I20260926 16:38:01.373517 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:01.354731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.10.110.62:33509)
0926 16:38:01.355430 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:01.355442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:01.355482 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:38:01.357080 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:01.357101 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:01.357469 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:01.357802 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:01.359225 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:01.360587 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:01.361792 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:01.362450 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:01.362612 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:01.366226 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:01.366234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:01.366252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:01.366286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:01.369329 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:01.370030 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:01.370037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:01.370043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:01.370226 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:01.370626 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:01.370631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:01.370634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:01.371183 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:01.371466 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:01.371816 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 16:38:01.372263 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":282,"threads_started":1}
I20260926 16:38:01.380692 18967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:01.381053 18967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:01.381127 18967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89decb19e8e94bd49e9f7f1bb6a8fd0f, State: Initialized, Role: FOLLOWER
I20260926 16:38:01.381623 18967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:01.381810 18967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:38:01.381901 18967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:38:01.382016 18967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:01.384079 18967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:01.384528 18967 leader_election.cc:302] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89decb19e8e94bd49e9f7f1bb6a8fd0f; no voters:
I20260926 16:38:01.385113 18967 leader_election.cc:288] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:38:01.385251 18972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:38:01.385803 18972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 LEADER]: Becoming Leader. State: Replica: 89decb19e8e94bd49e9f7f1bb6a8fd0f, State: Running, Role: LEADER
I20260926 16:38:01.386420 18972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:01.387549 18967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:38:01.389333 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } } }
I20260926 16:38:01.389684 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:01.390370 18973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89decb19e8e94bd49e9f7f1bb6a8fd0f. Latest consensus state: current_term: 1 leader_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } } }
I20260926 16:38:01.390724 18973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:01.392709 18978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:38:01.397217 18978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:38:01.406217 18978 catalog_manager.cc:1347] Generated new cluster ID: 27d768e086134d779771332f770afcaf
I20260926 16:38:01.406344 18978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:38:01.427428 18978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:38:01.428833 18978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:38:01.436862 18978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Generated new TSK 0
I20260926 16:38:01.437740 18978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:38:01.496362 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33509
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:01.816236 18993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:01.816538 18993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:01.816670 18993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:01.824062 18993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:01.824178 18993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:01.824332 18993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:38:01.833829 18993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:01.835114 18993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:01.836484 18993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:01.849331 18998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:01.849392 18999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:01.849773 19001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:01.850119 18993 server_base.cc:1034] running on GCE node
I20260926 16:38:01.850692 18993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:01.851658 18993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:01.852887 18993 hybrid_clock.cc:648] HybridClock initialized: now 1790440681852826 us; error 49 us; skew 500 ppm
I20260926 16:38:01.853199 18993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:01 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440681, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:01.861339 18993 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:38:01.862884 18993 webserver.cc:460] Webserver started at http://127.10.110.1:45689/ using document root <none> and password file <none>
I20260926 16:38:01.863584 18993 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:01.863700 18993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:01.864045 18993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:01.866953 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "97b6770e72ea41cf9122ee75c73442ff"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "2ecb67909ed694cda8937dc792c12901"
server_key_iv: "6c0f24c6d87e4862f0a9ef3969c8fb4f"
server_key_version: "encryptionkey@0"
I20260926 16:38:01.867702 18993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "97b6770e72ea41cf9122ee75c73442ff"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "2ecb67909ed694cda8937dc792c12901"
server_key_iv: "6c0f24c6d87e4862f0a9ef3969c8fb4f"
server_key_version: "encryptionkey@0"
I20260926 16:38:01.873090 18993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:38:01.876376 19008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:01.877609 18993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:38:01.877754 18993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "97b6770e72ea41cf9122ee75c73442ff"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "2ecb67909ed694cda8937dc792c12901"
server_key_iv: "6c0f24c6d87e4862f0a9ef3969c8fb4f"
server_key_version: "encryptionkey@0"
I20260926 16:38:01.877939 18993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:01.905741 18993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:01.909662 18993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:01.909945 18993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:01.910929 18993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:01.912325 18993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:01.912402 18993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:01.912518 18993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:01.912572 18993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:01.942548 18993 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:42585
I20260926 16:38:01.942579 19120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:42585 every 8 connection(s)
I20260926 16:38:01.943778 18993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:38:01.949014 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 18993
I20260926 16:38:01.949213 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:38:01.949988 10680 external_mini_cluster.cc:1428] Setting key 04e14dbab4fcbee782b957edb8eb032b
Sep 26 16:38:01 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440681, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:01.965431 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:01.946151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:54281 (local address 127.10.110.62:33509)
0926 16:38:01.946549 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:01.946554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:01.947465 (+ 911us) server_negotiation.cc:407] Connection header received
0926 16:38:01.949551 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:01.949555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:01.949627 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:01.949719 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:01.951421 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:01.952294 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:01.954448 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:01.954813 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:01.954912 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:01.959203 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:01.959210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:01.959215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:01.959259 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:01.962294 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:01.963082 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:01.963089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:01.963093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:01.963218 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:01.963666 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:01.963671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:01.963675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:01.963996 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:01.964127 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:01.964857 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 16:38:01.965080 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":192,"threads_started":1}
I20260926 16:38:01.966542 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:01.946690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:54281)
0926 16:38:01.947316 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:01.947358 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:01.949245 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:01.950002 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:01.950015 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:01.950707 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:01.951266 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:01.951283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:01.952452 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:01.952458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:01.954307 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:01.954321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:01.955010 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:01.955016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:01.955279 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:01.956048 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:01.956075 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:01.958891 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:01.962471 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:01.962487 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:01.962503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:01.962957 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:01.963354 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:01.963359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:01.963363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:01.963556 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:01.964114 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:01.964123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:01.964484 (+ 361us) client_negotiation.cc:769] Sending connection context
0926 16:38:01.964727 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 16:38:01.965272 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":240,"threads_started":1}
I20260926 16:38:01.968112 19121 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33509
I20260926 16:38:01.968431 19121 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:01.969136 19121 heartbeater.cc:511] Master 127.10.110.62:33509 requested a full tablet report, sending...
I20260926 16:38:01.971189 18932 ts_manager.cc:194] Registered new tserver with Master: 97b6770e72ea41cf9122ee75c73442ff (127.10.110.1:42585)
I20260926 16:38:01.972653 18932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:54281
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:38:02.010387 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33509
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:02.328333 19127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:02.328605 19127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:02.328697 19127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:02.335740 19127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:02.335862 19127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:02.336002 19127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:38:02.345512 19127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:02.346736 19127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:02.348017 19127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:02.360481 19133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:02.360610 19132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:02.360972 19135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:02.361603 19127 server_base.cc:1034] running on GCE node
I20260926 16:38:02.362228 19127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:02.363147 19127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:02.364337 19127 hybrid_clock.cc:648] HybridClock initialized: now 1790440682364282 us; error 48 us; skew 500 ppm
I20260926 16:38:02.364627 19127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:02 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440682, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:02.367664 19127 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:38:02.369091 19127 webserver.cc:460] Webserver started at http://127.10.110.2:33365/ using document root <none> and password file <none>
I20260926 16:38:02.369881 19127 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:02.370020 19127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:02.370347 19127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:02.374135 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "632ef123e5d54d8d893f013804e1d9ad"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "76bdc6cf9d33cfb8d4beadc747441062"
server_key_iv: "72d089b8c48d7690b9bdaab7237cab6e"
server_key_version: "encryptionkey@0"
I20260926 16:38:02.375123 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "632ef123e5d54d8d893f013804e1d9ad"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "76bdc6cf9d33cfb8d4beadc747441062"
server_key_iv: "72d089b8c48d7690b9bdaab7237cab6e"
server_key_version: "encryptionkey@0"
I20260926 16:38:02.380836 19127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 16:38:02.384244 19142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:02.385687 19127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:38:02.385866 19127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "632ef123e5d54d8d893f013804e1d9ad"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "76bdc6cf9d33cfb8d4beadc747441062"
server_key_iv: "72d089b8c48d7690b9bdaab7237cab6e"
server_key_version: "encryptionkey@0"
I20260926 16:38:02.386070 19127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:02.407239 19127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:02.411072 19127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:02.411325 19127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:02.412295 19127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:02.413602 19127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:02.413679 19127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:02.413800 19127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:02.413870 19127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:02.442677 19127 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:35755
I20260926 16:38:02.442725 19254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:35755 every 8 connection(s)
I20260926 16:38:02.443959 19127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:38:02.445931 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19127
I20260926 16:38:02.446079 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:38:02.446523 10680 external_mini_cluster.cc:1428] Setting key 5c97ece5b719e592fe9487ed6d6e3a48
Sep 26 16:38:02 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440682, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:02.463654 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:02.446610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:35257 (local address 127.10.110.62:33509)
0926 16:38:02.446809 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:02.446816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:02.448115 (+ 1299us) server_negotiation.cc:407] Connection header received
0926 16:38:02.449968 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:02.449973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:02.450054 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:02.450139 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:02.451590 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.452348 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.453855 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.454276 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.454385 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:02.457883 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:02.457888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:02.457892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:02.457924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:02.460428 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:02.461271 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:02.461277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:02.461280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:02.461432 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:02.461894 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:02.461897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:02.461899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:02.462158 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:02.462272 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:02.463202 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 16:38:02.463382 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:38:02.465147 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:02.447167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:35257)
0926 16:38:02.447963 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:02.448012 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:02.449697 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:02.450385 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:02.450394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:02.450892 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:02.451429 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:02.451449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.452488 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.452494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:02.453718 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:02.453729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.454484 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.454491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:02.454708 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:02.455330 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:02.455352 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:02.457620 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:02.460561 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:02.460570 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:02.460587 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:02.461155 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:02.461534 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:02.461538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:02.461542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:02.461787 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:02.462279 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:02.462288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:02.462765 (+ 477us) client_negotiation.cc:769] Sending connection context
0926 16:38:02.463101 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 16:38:02.463662 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":290,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:38:02.466650 19255 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33509
I20260926 16:38:02.466923 19255 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:38:02.467648 19255 heartbeater.cc:511] Master 127.10.110.62:33509 requested a full tablet report, sending...
I20260926 16:38:02.469159 18932 ts_manager.cc:194] Registered new tserver with Master: 632ef123e5d54d8d893f013804e1d9ad (127.10.110.2:35755)
I20260926 16:38:02.469921 18932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:35257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:38:02.494344 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33509
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:02.803681 19260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:02.804008 19260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:02.804147 19260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:02.811422 19260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:02.811566 19260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:02.811741 19260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:38:02.821893 19260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:02.823196 19260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:02.824561 19260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:02.836870 19266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:02.836890 19268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:02.836827 19265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:02.837672 19260 server_base.cc:1034] running on GCE node
I20260926 16:38:02.838253 19260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:02.839154 19260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:02.840358 19260 hybrid_clock.cc:648] HybridClock initialized: now 1790440682840309 us; error 51 us; skew 500 ppm
I20260926 16:38:02.840659 19260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:02 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440682, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:02.843550 19260 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:38:02.844970 19260 webserver.cc:460] Webserver started at http://127.10.110.3:44253/ using document root <none> and password file <none>
I20260926 16:38:02.845605 19260 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:02.845700 19260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:02.845978 19260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:02.848789 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "099be8f60f2e48d2b60fa371faf14164"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "997d329f1f9e96a7bf22908c7b88b6c2"
server_key_iv: "e05bf42a48f967cfaa09b805dcc1d16f"
server_key_version: "encryptionkey@0"
I20260926 16:38:02.849522 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "099be8f60f2e48d2b60fa371faf14164"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "997d329f1f9e96a7bf22908c7b88b6c2"
server_key_iv: "e05bf42a48f967cfaa09b805dcc1d16f"
server_key_version: "encryptionkey@0"
I20260926 16:38:02.854560 19260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:38:02.858070 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:02.859323 19260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:38:02.859467 19260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "099be8f60f2e48d2b60fa371faf14164"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "997d329f1f9e96a7bf22908c7b88b6c2"
server_key_iv: "e05bf42a48f967cfaa09b805dcc1d16f"
server_key_version: "encryptionkey@0"
I20260926 16:38:02.859654 19260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:02.882206 19260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:02.886343 19260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:02.886615 19260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:02.887633 19260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:02.889024 19260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:02.889099 19260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:02.889217 19260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:02.889266 19260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:02.917981 19260 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:37493
I20260926 16:38:02.918026 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:37493 every 8 connection(s)
I20260926 16:38:02.919181 19260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:38:02.923239 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19260
I20260926 16:38:02.923374 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:38:02.923784 10680 external_mini_cluster.cc:1428] Setting key b35718b535b4bc8d9508baa651a29ce8
Sep 26 16:38:02 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440682, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:02.937252 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:02.921507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:46787 (local address 127.10.110.62:33509)
0926 16:38:02.921709 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:02.921715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:02.922833 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 16:38:02.924686 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:02.924692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:02.924802 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:02.924893 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:02.926436 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.927180 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.928656 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.929045 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.929133 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:02.932320 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:02.932325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:02.932328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:02.932357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:02.934616 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:02.935297 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:02.935301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:02.935303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:02.935433 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:02.935758 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:02.935760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:02.935762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:02.936040 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:02.936148 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:02.936763 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 16:38:02.936966 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:38:02.938329 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:02.922016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:46787)
0926 16:38:02.922680 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:02.922717 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:02.924394 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:02.925131 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:02.925139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:02.925714 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:02.926274 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:02.926296 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.927297 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.927300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:02.928533 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:02.928545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:02.929201 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:02.929205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:02.929383 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:02.929973 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:02.929996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:02.932135 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:02.934767 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:02.934774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:02.934787 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:02.935188 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:02.935509 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:02.935512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:02.935514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:02.935679 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:02.936142 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:02.936148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:02.936455 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 16:38:02.936672 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 16:38:02.937079 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":244,"threads_started":1}
I20260926 16:38:02.939531 19388 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33509
I20260926 16:38:02.939810 19388 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:02.940526 19388 heartbeater.cc:511] Master 127.10.110.62:33509 requested a full tablet report, sending...
I20260926 16:38:02.942068 18932 ts_manager.cc:194] Registered new tserver with Master: 099be8f60f2e48d2b60fa371faf14164 (127.10.110.3:37493)
I20260926 16:38:02.943133 18932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:46787
I20260926 16:38:02.951529 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.1@KRBTEST.COM" created.
I20260926 16:38:02.975641 19121 heartbeater.cc:503] Master 127.10.110.62:33509 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab.
Entry for principal oryx/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab.
WARNING: no policy specified for oryx/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab.
Entry for principal oryx/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab.
WARNING: no policy specified for oryx/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab.
Entry for principal oryx/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab.
I20260926 16:38:03.074198 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 18993
Sep 26 16:38:03 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:38:03.092149 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:03.075298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.10.110.1:42585)
0926 16:38:03.075975 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:03.075983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:03.076010 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:38:03.076073 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:03.076080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:03.076295 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:03.076428 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:03.077796 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:03.078793 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:03.080168 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:03.081120 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:03.081270 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:03.085978 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:03.085990 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:03.086006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:03.086043 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:03.089210 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:03.090004 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:03.090013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:03.090015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:03.090148 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:03.090498 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:03.090503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:03.090507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:03.090888 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:03.091179 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:03.091491 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 16:38:03.091753 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":347,"threads_started":1}
I20260926 16:38:03.093515 19116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:03.174144 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 18993
I20260926 16:38:03.188920 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19127
Sep 26 16:38:03 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:38:03.203807 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:03.189677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.10.110.2:35755)
0926 16:38:03.190189 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:03.190197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:03.190222 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:38:03.190280 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:03.190286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:03.190490 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:03.190610 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:03.191742 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:03.193318 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:03.194662 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:03.195187 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:03.195285 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:03.199063 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:03.199073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:03.199087 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:03.199126 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:03.201575 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:03.202147 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:03.202154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:03.202156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:03.202274 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:03.202536 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:03.202539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:03.202541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:03.202848 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:03.203069 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:03.203272 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 16:38:03.203461 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260926 16:38:03.205056 19250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:03.285104 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19127
I20260926 16:38:03.297843 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19260
Sep 26 16:38:03 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:38:03.314177 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:03.298907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.10.110.3:37493)
0926 16:38:03.299505 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:03.299512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:03.299541 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:38:03.299596 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:03.299604 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:03.299812 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:03.299977 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:03.301261 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:03.302278 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:03.303590 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:03.304310 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:03.304416 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:03.308665 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:03.308677 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:03.308693 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:03.308737 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:03.311382 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:03.312151 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:03.312158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:03.312161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:03.312304 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:03.312681 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:03.312685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:03.312687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:03.313016 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:03.313326 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:03.313571 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 16:38:03.313853 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":285,"threads_started":1}
I20260926 16:38:03.315449 19383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:03.397681 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19260
I20260926 16:38:03.410018 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 18900
I20260926 16:38:03.411139 18962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:03.512207 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 18900
I20260926 16:38:03.528324 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33509
--webserver_interface=127.10.110.62
--webserver_port=38367
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:03.837807 19405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:03.838099 19405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:03.838212 19405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:03.845353 19405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:38:03.845463 19405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:03.845534 19405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:03.845610 19405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:38:03.845670 19405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:38:03.855338 19405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:03.856628 19405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:03.857892 19405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:03.868774 19410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:03.868798 19411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:03.868938 19413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:03.869489 19405 server_base.cc:1034] running on GCE node
I20260926 16:38:03.870244 19405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:03.871639 19405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:03.872846 19405 hybrid_clock.cc:648] HybridClock initialized: now 1790440683872776 us; error 60 us; skew 500 ppm
I20260926 16:38:03.873178 19405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:03 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440683, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:03.876000 19405 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:38:03.877349 19405 webserver.cc:460] Webserver started at http://127.10.110.62:38367/ using document root <none> and password file <none>
I20260926 16:38:03.877982 19405 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:03.878100 19405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:03.883162 19405 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:38:03.885797 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:03.887006 19405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:38:03.887151 19405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "6cc399f75b4584de5f96112c51f53b41"
server_key_iv: "6eef72c80ec521a27e1356c4d05e57c1"
server_key_version: "encryptionkey@0"
I20260926 16:38:03.888113 19405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:03.912433 19405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:03.916124 19405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:03.916356 19405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:03.932718 19405 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33509
I20260926 16:38:03.932721 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33509 every 8 connection(s)
I20260926 16:38:03.934127 19405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:38:03.936650 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19405
I20260926 16:38:03.937381 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:42585
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:33509
--webserver_port=45689
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:38:03.940778 19472 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:38:03.944103 19472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Bootstrap starting.
I20260926 16:38:03.969888 19472 log.cc:824] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:03.980974 19472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:38:03.981604 19472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Bootstrap complete.
I20260926 16:38:03.994511 19472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:03.995633 19472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89decb19e8e94bd49e9f7f1bb6a8fd0f, State: Initialized, Role: FOLLOWER
I20260926 16:38:03.996398 19472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:03.996681 19472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:38:03.996797 19472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:38:03.996968 19472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:38:04.002679 19472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:04.003281 19472 leader_election.cc:302] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89decb19e8e94bd49e9f7f1bb6a8fd0f; no voters:
I20260926 16:38:04.003923 19472 leader_election.cc:288] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:38:04.004065 19476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:38:04.005172 19476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [term 2 LEADER]: Becoming Leader. State: Replica: 89decb19e8e94bd49e9f7f1bb6a8fd0f, State: Running, Role: LEADER
I20260926 16:38:04.005667 19476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } }
I20260926 16:38:04.005980 19472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:38:04.007880 19478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89decb19e8e94bd49e9f7f1bb6a8fd0f. Latest consensus state: current_term: 2 leader_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } } }
I20260926 16:38:04.007885 19477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89decb19e8e94bd49e9f7f1bb6a8fd0f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33509 } } }
I20260926 16:38:04.008224 19478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:04.008221 19477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:04.020308 19482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:38:04.027305 19482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:38:04.028733 19482 catalog_manager.cc:1259] Loaded cluster ID: 27d768e086134d779771332f770afcaf
I20260926 16:38:04.028884 19482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:38:04.032668 19482 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:38:04.034080 19482 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89decb19e8e94bd49e9f7f1bb6a8fd0f: Loaded TSK: 0
I20260926 16:38:04.035626 19482 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:38:04.284888 19474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:04.285177 19474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:04.285290 19474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:04.292721 19474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:04.292831 19474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:04.292972 19474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:38:04.302799 19474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.1.keytab
--principal=oryx/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:42585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=45689
--tserver_master_addrs=127.10.110.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:04.304129 19474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:04.305382 19474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:04.319702 19498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:04.319730 19499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:04.320145 19501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:04.320550 19474 server_base.cc:1034] running on GCE node
I20260926 16:38:04.321189 19474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:04.322165 19474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:04.323393 19474 hybrid_clock.cc:648] HybridClock initialized: now 1790440684323326 us; error 56 us; skew 500 ppm
I20260926 16:38:04.323760 19474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:04 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440684, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:04.327015 19474 init.cc:377] Logged in from keytab as oryx/127.10.110.1@KRBTEST.COM (short username oryx)
I20260926 16:38:04.328617 19474 webserver.cc:460] Webserver started at http://127.10.110.1:45689/ using document root <none> and password file <none>
I20260926 16:38:04.329346 19474 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:04.329475 19474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:04.334975 19474 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:38:04.337711 19508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:04.339078 19474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:38:04.339257 19474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "97b6770e72ea41cf9122ee75c73442ff"
format_stamp: "Formatted at 2026-09-26 16:38:01 on dist-test-slave-whjp"
server_key: "2ecb67909ed694cda8937dc792c12901"
server_key_iv: "6c0f24c6d87e4862f0a9ef3969c8fb4f"
server_key_version: "encryptionkey@0"
I20260926 16:38:04.340284 19474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:04.358664 19474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:04.362432 19474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:04.362675 19474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:04.363588 19474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:04.364853 19474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:04.364943 19474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:04.365051 19474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:04.365123 19474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:04.396327 19474 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:42585
I20260926 16:38:04.396374 19620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:42585 every 8 connection(s)
I20260926 16:38:04.397648 19474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:38:04.400529 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19474
I20260926 16:38:04.401221 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:35755
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:33509
--webserver_port=33365
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:38:04 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:04 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:38:04.796943 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:04.797207 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:04.797322 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:04.804478 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:04.804589 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:04.804744 19625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:38:04.814364 19625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.2.keytab
--principal=oryx/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:35755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=33365
--tserver_master_addrs=127.10.110.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:04.815665 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:04.817013 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:04.832187 19632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:04.832374 19631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:04.832757 19634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:04.833323 19625 server_base.cc:1034] running on GCE node
I20260926 16:38:04.833935 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:04.834900 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:04.836119 19625 hybrid_clock.cc:648] HybridClock initialized: now 1790440684836024 us; error 80 us; skew 500 ppm
I20260926 16:38:04.836570 19625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:04 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440684, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:04.840150 19625 init.cc:377] Logged in from keytab as oryx/127.10.110.2@KRBTEST.COM (short username oryx)
I20260926 16:38:04.841794 19625 webserver.cc:460] Webserver started at http://127.10.110.2:33365/ using document root <none> and password file <none>
I20260926 16:38:04.842504 19625 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:04.842633 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:04.848716 19625 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:38:04.851591 19641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:04.853175 19625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:38:04.853335 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "632ef123e5d54d8d893f013804e1d9ad"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "76bdc6cf9d33cfb8d4beadc747441062"
server_key_iv: "72d089b8c48d7690b9bdaab7237cab6e"
server_key_version: "encryptionkey@0"
I20260926 16:38:04.854254 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:04.867980 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:04.871712 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:04.871978 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:04.872960 19625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:04.874267 19625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:04.874368 19625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:04.874464 19625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:04.874528 19625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:04.910557 19625 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:35755
I20260926 16:38:04.910614 19753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:35755 every 8 connection(s)
I20260926 16:38:04.911985 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:38:04.913221 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19625
I20260926 16:38:04.913771 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:37493
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:33509
--webserver_port=44253
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:38:04 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:04 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:38:05.451130 19757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:05.451514 19757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:05.451705 19757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:05.463275 19757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:05.463431 19757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:05.463611 19757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:38:05.479242 19757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/krb5kdc/oryx_127.10.110.3.keytab
--principal=oryx/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:37493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=44253
--tserver_master_addrs=127.10.110.62:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:05.481006 19757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:05.482878 19757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:05.504442 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:05.504415 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:05.507805 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:05.508795 19757 server_base.cc:1034] running on GCE node
I20260926 16:38:05.509666 19757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:05.511036 19757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:05.512311 19757 hybrid_clock.cc:648] HybridClock initialized: now 1790440685512242 us; error 53 us; skew 500 ppm
I20260926 16:38:05.512737 19757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440685, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:05.517264 19757 init.cc:377] Logged in from keytab as oryx/127.10.110.3@KRBTEST.COM (short username oryx)
I20260926 16:38:05.519351 19757 webserver.cc:460] Webserver started at http://127.10.110.3:44253/ using document root <none> and password file <none>
I20260926 16:38:05.520453 19757 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:05.520618 19757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:05.528815 19757 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:38:05.532553 19774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:05.534401 19757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:38:05.534597 19757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "099be8f60f2e48d2b60fa371faf14164"
format_stamp: "Formatted at 2026-09-26 16:38:02 on dist-test-slave-whjp"
server_key: "997d329f1f9e96a7bf22908c7b88b6c2"
server_key_iv: "e05bf42a48f967cfaa09b805dcc1d16f"
server_key_version: "encryptionkey@0"
I20260926 16:38:05.536191 19757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260926 16:38:05.545130 19624 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.134s user 0.004s sys 0.007s
I20260926 16:38:05.546893 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:04.400462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:48057)
0926 16:38:04.401257 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:04.401526 (+ 269us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:04.403678 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:04.405377 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:04.405387 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:04.406080 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:04.406807 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:04.406830 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.408460 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.408464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:04.409620 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:04.409631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.410599 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.410606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:04.410849 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:04.411442 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:04.411462 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:05.545830 (+1134368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":310,"threads_started":1}
W20260926 16:38:05.546958 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:04.400357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48057 (local address 127.10.110.62:33509)
0926 16:38:04.404263 (+ 3906us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:04.404282 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:04.404319 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:38:04.404593 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:04.404607 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:04.404909 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:04.405178 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:04.407002 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.408312 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.409755 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.410391 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.410547 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.546236 (+1135689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:48057: BlockingRecv error: recv got EOF from 127.10.110.1:48057 (error 108)
Metrics: {"server-negotiator.queue_time_us":3695,"thread_start_us":3503,"threads_started":1}
W20260926 16:38:05.548142 19621 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:33509 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.110.62:33509: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:05.556684 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:05.548772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:43311)
0926 16:38:05.548940 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:05.548958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:05.549053 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:05.549379 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:05.549382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:05.549708 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:05.549866 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.549874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.550859 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.550864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.551782 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.551794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.552476 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.552483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.552627 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.553171 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:05.553188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:05.556399 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:38:05.556818 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:05.548859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:43311 (local address 127.10.110.62:33509)
0926 16:38:05.549044 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:05.549050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:05.549077 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:38:05.549150 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:05.549156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:05.549241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:05.549371 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:05.550021 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.550705 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.551928 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.552307 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.552410 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.556597 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:43311: BlockingRecv error: recv got EOF from 127.10.110.1:43311 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:05.565059 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:05.557513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:48647)
0926 16:38:05.557649 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:05.557667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:05.557759 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:05.558192 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:05.558196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:05.558481 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:05.558620 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.558627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.559603 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.559607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.560554 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.560566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.561208 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.561214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.561363 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.561883 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:05.561899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:05.564806 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 16:38:05.565209 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:05.557798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48647 (local address 127.10.110.62:33509)
0926 16:38:05.557943 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:05.557948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:05.557963 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:38:05.558005 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:05.558008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:05.558075 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:05.558156 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:05.558746 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.559472 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.560677 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.561081 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.561180 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.565009 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:48647: BlockingRecv error: recv got EOF from 127.10.110.1:48647 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 16:38:05.565548 19621 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:38:05.566100 19757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:05.571991 19757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:05.572311 19757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:05.573589 19757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:05.575440 19757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:05.575560 19757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:05.575678 19757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:05.575758 19757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:05.611600 19757 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:37493
I20260926 16:38:05.611614 19886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:37493 every 8 connection(s)
I20260926 16:38:05.613587 19757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:38:05.618708 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19757
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:05.648623 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:05.619695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.10.110.62:33509)
0926 16:38:05.630462 (+ 10767us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:05.630469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:05.630489 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:38:05.630544 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:05.630549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:05.630627 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:05.630762 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:05.632891 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.634099 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.636096 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.636728 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.636856 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.640526 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:05.640538 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:05.640566 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:05.640613 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:05.644484 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:05.645449 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:05.645460 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:05.645471 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:05.645734 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:05.646115 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:05.646120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:05.646124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:05.646849 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:05.647211 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:05.647665 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 16:38:05.648136 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10629,"thread_start_us":9616,"threads_started":1}
W20260926 16:38:05.932669 19756 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.005s user 0.003s sys 0.010s
W20260926 16:38:05.933940 19758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:04.919249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:56737 (local address 127.10.110.62:33509)
0926 16:38:04.919742 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:04.919748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:04.919768 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:38:04.922013 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:04.922019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:04.922096 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:04.922186 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:04.923637 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.924430 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.925910 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.926302 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.926414 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.933614 (+1007200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:56737: BlockingRecv error: recv got EOF from 127.10.110.2:56737 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":1161,"threads_started":1}
I20260926 16:38:05.934449 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:04.918644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:56737)
0926 16:38:04.919520 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:04.919583 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:04.921734 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:04.922429 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:04.922437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:04.922940 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:04.923481 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:04.923498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.924568 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.924574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:04.925783 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:04.925794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:04.926506 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:04.926514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:04.926736 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:04.927352 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:04.927373 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:05.933392 (+1006019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":614,"threads_started":1}
W20260926 16:38:05.935545 19754 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:33509 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.110.62:33509: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:05.945055 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:05.936162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:34849)
0926 16:38:05.936336 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:05.936356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:05.936464 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:05.936744 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:05.936749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:05.937094 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:05.937244 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.937253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.938305 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.938310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.939214 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.939225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.940285 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.940291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.940424 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.941081 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:05.941102 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:05.944779 (+ 3677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:38:05.945261 19758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:05.936264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:34849 (local address 127.10.110.62:33509)
0926 16:38:05.936413 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:05.936418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:05.936435 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:38:05.936562 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:05.936568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:05.936628 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:05.936704 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:05.937383 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.938165 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.939340 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.939960 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.940067 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.944989 (+ 4922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:34849: BlockingRecv error: recv got EOF from 127.10.110.2:34849 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:05 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:05.955657 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:05.945854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:57289)
0926 16:38:05.945989 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:05.946007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:05.946104 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:05.946688 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:05.946694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:05.947158 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:05.947399 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.947413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.948447 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.948452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.950161 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.950179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.950868 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.950874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.951091 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.951725 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:05.951750 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:05.955390 (+ 3640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 16:38:05.955865 19758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:05.946249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:57289 (local address 127.10.110.62:33509)
0926 16:38:05.946395 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:05.946399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:05.946412 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:38:05.946450 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:05.946453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:05.946504 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:05.946572 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:05.947555 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.948322 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.950331 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.950722 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.950826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.955578 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:57289: BlockingRecv error: recv got EOF from 127.10.110.2:57289 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 16:38:05.956071 19754 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:06.575975 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:06.566217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:48157)
0926 16:38:06.566765 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:06.566790 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:06.566930 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:06.567357 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:06.567362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:06.567859 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:06.568033 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.568044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.569074 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.569080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.570081 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.570093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.570856 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.570863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.571027 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.571648 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:06.571668 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:06.575593 (+ 3925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
W20260926 16:38:06.576117 19894 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:06.566307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48157 (local address 127.10.110.62:33509)
0926 16:38:06.567046 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:06.567052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:06.567071 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:38:06.567122 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:06.567126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:06.567185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:06.567282 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:06.568186 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.568926 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.570226 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.570653 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.570775 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.575861 (+ 5086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:48157: BlockingRecv error: recv got EOF from 127.10.110.1:48157 (error 108)
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":497,"threads_started":1}
W20260926 16:38:06.628314 19889 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.999s user 0.005s sys 0.009s
W20260926 16:38:06.629359 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:05.616266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:54459 (local address 127.10.110.62:33509)
0926 16:38:05.616428 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:05.616433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:05.618002 (+ 1569us) server_negotiation.cc:407] Connection header received
0926 16:38:05.620441 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:05.620449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:05.620542 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:05.620614 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:05.623130 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.624327 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.626923 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.627665 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.627763 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.629132 (+1001369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:54459: BlockingRecv error: recv got EOF from 127.10.110.3:54459 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:38:06.629830 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:05.617055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:54459)
0926 16:38:05.617849 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:05.617890 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:05.620053 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:05.621273 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:05.621286 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:05.622065 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:05.622904 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.622928 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.624565 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.624570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.626732 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:05.626749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:05.628168 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:05.628175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:05.628530 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:05.629452 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:05.629490 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:06.628976 (+999486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":322,"threads_started":1}
W20260926 16:38:06.630558 19887 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:33509 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.110.62:33509: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:06.639065 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:06.631111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:42593)
0926 16:38:06.631272 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:06.631293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:06.631400 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:06.631700 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:06.631704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:06.632088 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:06.632244 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.632252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.633241 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.633247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.634184 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.634195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.634860 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.634866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.635017 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.635533 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:06.635549 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:06.638783 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:38:06.639238 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:06.631186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:42593 (local address 127.10.110.62:33509)
0926 16:38:06.631409 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:06.631415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:06.631433 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:38:06.631526 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:06.631531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:06.631589 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:06.631666 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:06.632377 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.633116 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.634313 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.634705 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.634809 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.639015 (+ 4206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:42593: BlockingRecv error: recv got EOF from 127.10.110.3:42593 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:06.647552 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:06.639742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:40765)
0926 16:38:06.639912 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:06.639938 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:06.640022 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:06.640280 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:06.640282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:06.640551 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:06.640699 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.640707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.641726 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.641730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.642747 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.642759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.643412 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.643417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.643553 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.644106 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:06.644123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:06.647353 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 16:38:06.647753 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:06.639807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:40765 (local address 127.10.110.62:33509)
0926 16:38:06.639966 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:06.639970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:06.639986 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:38:06.640109 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:06.640113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:06.640170 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:06.640247 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:06.640842 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.641617 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.642878 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.643285 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.643386 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.647541 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:40765: BlockingRecv error: recv got EOF from 127.10.110.3:40765 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 16:38:06.648068 19887 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:06 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:06.969156 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:06.956765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:44907)
0926 16:38:06.957348 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:06.957389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:06.957520 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:06.957866 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:06.957869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:06.958360 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:06.958541 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.958552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.959584 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.959589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.960672 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:06.960685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.961488 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.961492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:06.961618 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.965499 (+ 3881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:06.965519 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:06.968845 (+ 3326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
W20260926 16:38:06.969401 19623 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:06.957018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:44907 (local address 127.10.110.62:33509)
0926 16:38:06.957193 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:06.957199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:06.957474 (+ 275us) server_negotiation.cc:407] Connection header received
0926 16:38:06.957643 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:06.957647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:06.957730 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:06.957808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:06.958703 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.959425 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.960825 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:06.961269 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:06.961367 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:06.969091 (+ 7724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:44907: BlockingRecv error: recv got EOF from 127.10.110.2:44907 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 16:38:07 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:07 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:07.586571 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:07.576964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:53881)
0926 16:38:07.577476 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:07.577500 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:07.577627 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:07.578163 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:07.578166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:07.578598 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:07.578812 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:07.578822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.579893 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.579897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:07.580820 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:07.580832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.581520 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.581524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:07.581641 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:07.582287 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:07.582307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:07.586245 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
W20260926 16:38:07.586699 19897 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:07.577102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:53881 (local address 127.10.110.62:33509)
0926 16:38:07.577888 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:07.577893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:07.577914 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:38:07.577962 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:07.577965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:07.578020 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:07.578104 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:07.578944 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.579718 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.580941 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.581386 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.581483 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:07.586453 (+ 4970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:53881: BlockingRecv error: recv got EOF from 127.10.110.1:53881 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":502,"threads_started":1}
Sep 26 16:38:07 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:07 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:07.657465 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:07.648687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:36723)
0926 16:38:07.649257 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:07.649273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:07.649403 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:07.649737 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:07.649742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:07.650181 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:07.650361 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:07.650369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.651397 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.651401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:07.652387 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:07.652399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.653106 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.653112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:07.653261 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:07.653796 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:07.653812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:07.657148 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":293,"threads_started":1}
W20260926 16:38:07.657589 19897 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:07.648824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:36723 (local address 127.10.110.62:33509)
0926 16:38:07.649030 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:07.649037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:07.649351 (+ 314us) server_negotiation.cc:407] Connection header received
0926 16:38:07.649515 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:07.649521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:07.649602 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:07.649682 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:07.650492 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.651268 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.652509 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.652946 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.653049 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:07.657343 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:36723: BlockingRecv error: recv got EOF from 127.10.110.3:36723 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 16:38:07 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:07 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:07.978955 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:07.970216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:39161)
0926 16:38:07.970670 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:07.970686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:07.970807 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:07.971098 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:07.971101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:07.971532 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:07.971711 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:07.971722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.972801 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.972806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:07.973753 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:07.973767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.974512 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.974517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:07.974664 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:07.975232 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:07.975252 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:07.978674 (+ 3422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
W20260926 16:38:07.979136 19897 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:07.970289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:39161 (local address 127.10.110.62:33509)
0926 16:38:07.970467 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:07.970471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:07.970752 (+ 281us) server_negotiation.cc:407] Connection header received
0926 16:38:07.970914 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:07.970917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:07.970977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:07.971058 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:07.971897 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.972638 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.973901 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:07.974344 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:07.974441 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:07.978883 (+ 4442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:39161: BlockingRecv error: recv got EOF from 127.10.110.2:39161 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 16:38:08 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:08 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:08.595935 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:08.587474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:49447)
0926 16:38:08.588058 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:08.588078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:08.588215 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:08.588526 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:08.588531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:08.588906 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:08.589064 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:08.589073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.590087 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.590092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:08.590986 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:08.590997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.591722 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.591736 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:08.591937 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:08.592459 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:08.592477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:08.595616 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":276,"threads_started":1}
W20260926 16:38:08.596098 19901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:08.587647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:49447 (local address 127.10.110.62:33509)
0926 16:38:08.588152 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:08.588158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:08.588175 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:38:08.588328 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:08.588333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:08.588400 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:08.588481 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:08.589200 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.589933 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.591132 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.591593 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.591697 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:08.595868 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:49447: BlockingRecv error: recv got EOF from 127.10.110.1:49447 (error 108)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":305,"threads_started":1}
Sep 26 16:38:08 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:08 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:08.666644 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:08.658283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:39629)
0926 16:38:08.658924 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:08.658940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:08.659065 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:08.659371 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:08.659375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:08.659780 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:08.659979 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:08.659988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.660957 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.660963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:08.661891 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:08.661902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.662563 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.662569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:08.662748 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:08.663250 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:08.663267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:08.666403 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":401,"threads_started":1}
W20260926 16:38:08.666765 19901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:08.658405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:39629 (local address 127.10.110.62:33509)
0926 16:38:08.658568 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:08.658574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:08.659021 (+ 447us) server_negotiation.cc:407] Connection header received
0926 16:38:08.659177 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:08.659182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:08.659251 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:08.659328 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:08.660118 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.660830 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.662022 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.662429 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.662531 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:08.666576 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:39629: BlockingRecv error: recv got EOF from 127.10.110.3:39629 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 16:38:08 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:08 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:08.989336 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:08.980173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:56445)
0926 16:38:08.980701 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:08.980732 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:08.980868 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:08.981229 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:08.981233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:08.981583 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:08.981759 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:08.981769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.982837 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.982842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:08.983810 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:08.983863 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.984705 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.984710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:08.984855 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:08.985437 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:08.985455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:08.989004 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":267,"threads_started":1}
W20260926 16:38:08.989457 19901 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:08.980382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:56445 (local address 127.10.110.62:33509)
0926 16:38:08.980614 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:08.980620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:08.980852 (+ 232us) server_negotiation.cc:407] Connection header received
0926 16:38:08.980999 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:08.981003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:08.981071 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:08.981171 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:08.981914 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.982669 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.984037 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:08.984534 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:08.984646 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:08.989224 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:56445: BlockingRecv error: recv got EOF from 127.10.110.2:56445 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 16:38:09 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:09 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:09.605525 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:09.596977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:50869)
0926 16:38:09.597519 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:09.597535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:09.597650 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:09.597981 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:09.597983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:09.598360 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:09.598525 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:09.598533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.599522 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.599526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:09.600536 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:09.600548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.601354 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.601361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:09.601542 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:09.602194 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:09.602218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:09.605178 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
W20260926 16:38:09.605644 19905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:09.597078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:50869 (local address 127.10.110.62:33509)
0926 16:38:09.597563 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:09.597568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:09.597591 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:38:09.597788 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:09.597792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:09.597857 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:09.597957 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:09.598657 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.599394 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.600683 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.601157 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.601278 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:09.605427 (+ 4149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:50869: BlockingRecv error: recv got EOF from 127.10.110.1:50869 (error 108)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
Sep 26 16:38:09 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:09 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:09.676404 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:09.667587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:60653)
0926 16:38:09.668155 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:09.668172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:09.668289 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:09.668574 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:09.668577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:09.668949 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:09.669120 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:09.669129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.670195 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.670198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:09.671089 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:09.671100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.671766 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.671769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:09.671951 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:09.672549 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:09.672572 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:09.676175 (+ 3603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":291,"threads_started":1}
W20260926 16:38:09.676813 19905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:09.667699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:60653 (local address 127.10.110.62:33509)
0926 16:38:09.667982 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:09.667988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:09.668308 (+ 320us) server_negotiation.cc:407] Connection header received
0926 16:38:09.668375 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:09.668379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:09.668465 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:09.668548 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:09.669336 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.670078 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.671229 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.671653 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.671752 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:09.676395 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:60653: BlockingRecv error: recv got EOF from 127.10.110.3:60653 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Sep 26 16:38:09 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:09 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:09.998713 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:09.990330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:43595)
0926 16:38:09.990781 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:09.990797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:09.990904 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:09.991191 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:09.991194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:09.991590 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:09.991755 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:09.991764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.992881 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.992885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:09.993860 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:09.993872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.994540 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.994544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:09.994662 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:09.995204 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:09.995220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:09.998471 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
W20260926 16:38:09.998914 19905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:09.990360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:43595 (local address 127.10.110.62:33509)
0926 16:38:09.990545 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:09.990552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:09.990870 (+ 318us) server_negotiation.cc:407] Connection header received
0926 16:38:09.991003 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:09.991007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:09.991079 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:09.991154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:09.991914 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.992733 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.993993 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:09.994423 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:09.994525 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:09.998661 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:43595: BlockingRecv error: recv got EOF from 127.10.110.2:43595 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 16:38:10 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:10 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:10.615312 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:10.606460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:58143)
0926 16:38:10.606950 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:10.606968 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:10.607087 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:10.607418 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:10.607423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:10.607869 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:10.608041 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:10.608051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.608998 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.609002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:10.609993 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:10.610006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.610671 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.610676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:10.610836 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:10.611416 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:10.611435 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:10.615022 (+ 3587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
W20260926 16:38:10.615504 19909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:10.606496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:58143 (local address 127.10.110.62:33509)
0926 16:38:10.606938 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:10.606943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:10.607035 (+ 92us) server_negotiation.cc:407] Connection header received
0926 16:38:10.607180 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:10.607183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:10.607255 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:10.607373 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:10.608167 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.608873 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.610116 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.610530 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.610619 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:10.615282 (+ 4663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:58143: BlockingRecv error: recv got EOF from 127.10.110.1:58143 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
Sep 26 16:38:10 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:10 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:10.685902 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:10.677298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:32963)
0926 16:38:10.677801 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:10.677821 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:10.677954 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:10.678287 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:10.678290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:10.678625 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:10.678786 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:10.678795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.679796 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.679800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:10.680809 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:10.680820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.681505 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.681509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:10.681651 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:10.682237 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:10.682258 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:10.685638 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
W20260926 16:38:10.686043 19909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:10.677352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:32963 (local address 127.10.110.62:33509)
0926 16:38:10.677535 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:10.677541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:10.677906 (+ 365us) server_negotiation.cc:407] Connection header received
0926 16:38:10.678063 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:10.678067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:10.678125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:10.678199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:10.678917 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.679678 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.680941 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:10.681371 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:10.681466 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:10.685849 (+ 4383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:32963: BlockingRecv error: recv got EOF from 127.10.110.3:32963 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 16:38:11 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:11 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:11.016753 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:10.999713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:43323)
0926 16:38:11.000794 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:11.000814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:11.000937 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:11.001471 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:11.001477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:11.002005 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:11.002198 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:11.002210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.003657 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.003662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:11.005275 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:11.005290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.006166 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.006172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:11.006398 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:11.007000 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:11.007018 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:11.016336 (+ 9318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":618,"threads_started":1}
W20260926 16:38:11.016901 19909 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:11.000755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:43323 (local address 127.10.110.62:33509)
0926 16:38:11.000905 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:11.000911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:11.000932 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:38:11.001056 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:11.001061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:11.001153 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:11.001246 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:11.002350 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.003500 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.005435 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.006005 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.006101 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:11.016626 (+ 10525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:43323: BlockingRecv error: recv got EOF from 127.10.110.2:43323 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 16:38:11 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:11 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:11.625401 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:11.616229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:40329)
0926 16:38:11.616691 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:11.616710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:11.616824 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:11.617254 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:11.617257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:11.617744 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:11.617974 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:11.617983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.619016 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.619020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:11.620157 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:11.620170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.620887 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.620893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:11.621056 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:11.621756 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:11.621784 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:11.625173 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
W20260926 16:38:11.625689 19913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:11.616343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:40329 (local address 127.10.110.62:33509)
0926 16:38:11.616974 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:11.616978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:11.616994 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:38:11.617045 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:11.617049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:11.617116 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:11.617204 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:11.618116 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.618884 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.620283 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.620740 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.620849 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:11.625415 (+ 4566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:40329: BlockingRecv error: recv got EOF from 127.10.110.1:40329 (error 108)
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":430,"threads_started":1}
Sep 26 16:38:11 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:11 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:11.695611 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:11.686847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:58539)
0926 16:38:11.687319 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:11.687334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:11.687450 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:11.687797 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:11.687801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:11.688256 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:11.688429 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:11.688439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.689512 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.689516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:11.690455 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:11.690466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.691309 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.691314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:11.691475 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:11.692139 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:11.692163 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:11.695385 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
W20260926 16:38:11.695739 19913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:11.686950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:58539 (local address 127.10.110.62:33509)
0926 16:38:11.687162 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:11.687169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:11.687413 (+ 244us) server_negotiation.cc:407] Connection header received
0926 16:38:11.687543 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:11.687547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:11.687615 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:11.687730 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:11.688582 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.689377 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.690598 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:11.691164 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:11.691288 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:11.695562 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:58539: BlockingRecv error: recv got EOF from 127.10.110.3:58539 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:38:12 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:12 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:12.031275 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:12.017666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:35833)
0926 16:38:12.018100 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:12.018123 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:12.018230 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:12.018502 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:12.018505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:12.018888 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:12.019043 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:12.019052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.020097 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.020101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:12.021140 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:12.021155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.021771 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.021776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:12.021912 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:12.027892 (+ 5980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:12.027912 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:12.031048 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":227,"threads_started":1}
W20260926 16:38:12.031445 19913 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:12.017743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:35833 (local address 127.10.110.62:33509)
0926 16:38:12.017894 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:12.017898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:12.018182 (+ 284us) server_negotiation.cc:407] Connection header received
0926 16:38:12.018333 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:12.018337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:12.018396 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:12.018472 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:12.019147 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.019952 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.021250 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.021649 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.021731 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:12.031215 (+ 9484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:35833: BlockingRecv error: recv got EOF from 127.10.110.2:35833 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 16:38:12 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:12 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:12.635636 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:12.626443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:34053)
0926 16:38:12.627042 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:12.627059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:12.627167 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:12.627656 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:12.627659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:12.628090 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:12.628310 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:12.628321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.629550 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.629555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:12.630776 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:12.630789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.631485 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.631489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:12.631596 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:12.632176 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:12.632193 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:12.635390 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":283,"threads_started":1}
W20260926 16:38:12.635883 19917 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:12.626651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:34053 (local address 127.10.110.62:33509)
0926 16:38:12.627385 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:12.627389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:12.627408 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:38:12.627460 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:12.627463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:12.627529 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:12.627624 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:12.628475 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.629416 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.630930 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.631364 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.631468 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:12.635614 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:34053: BlockingRecv error: recv got EOF from 127.10.110.1:34053 (error 108)
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":485,"threads_started":1}
Sep 26 16:38:12 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:12 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:12.705596 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:12.696522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:34799)
0926 16:38:12.696989 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:12.697004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:12.697128 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:12.697458 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:12.697463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:12.697908 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:12.698068 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:12.698077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.699062 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.699067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:12.700065 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:12.700079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.700889 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.700895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:12.701067 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:12.701673 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:12.701691 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:12.705357 (+ 3666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
W20260926 16:38:12.705770 19917 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:12.696637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:34799 (local address 127.10.110.62:33509)
0926 16:38:12.696845 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:12.696851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:12.697080 (+ 229us) server_negotiation.cc:407] Connection header received
0926 16:38:12.697238 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:12.697242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:12.697334 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:12.697415 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:12.698207 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.698925 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.700187 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:12.700746 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:12.700861 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:12.705564 (+ 4703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:34799: BlockingRecv error: recv got EOF from 127.10.110.3:34799 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 16:38:13 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:13 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:13.040199 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:13.032145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:37931)
0926 16:38:13.032632 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:13.032648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:13.032778 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:13.033051 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:13.033055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:13.033429 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:13.033592 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:13.033601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.034547 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.034552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:13.035495 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:13.035506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.036188 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.036193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:13.036314 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:13.036827 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:13.036860 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:13.039948 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":226,"threads_started":1}
W20260926 16:38:13.040376 19917 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:13.032203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:37931 (local address 127.10.110.62:33509)
0926 16:38:13.032474 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:13.032480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:13.032714 (+ 234us) server_negotiation.cc:407] Connection header received
0926 16:38:13.032859 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:13.032863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:13.032932 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:13.033007 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:13.033708 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.034422 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.035616 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.036059 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.036143 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:13.040111 (+ 3968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:37931: BlockingRecv error: recv got EOF from 127.10.110.2:37931 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 26 16:38:13 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:13 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:13.646032 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:13.636591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:57297)
0926 16:38:13.637227 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:13.637241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:13.637365 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:13.637705 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:13.637710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:13.638109 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:13.638271 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:13.638279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.639348 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.639354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:13.640303 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:13.640315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.641131 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.641137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:13.641291 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:13.641958 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:13.641974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:13.645694 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":393,"threads_started":1}
W20260926 16:38:13.646344 19920 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:13.636609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:57297 (local address 127.10.110.62:33509)
0926 16:38:13.637086 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:13.637092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:13.637351 (+ 259us) server_negotiation.cc:407] Connection header received
0926 16:38:13.637504 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:13.637508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:13.637579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:13.637661 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:13.638424 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.639192 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.640439 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.640961 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.641100 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:13.645914 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:57297: BlockingRecv error: recv got EOF from 127.10.110.1:57297 (error 108)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":231,"threads_started":1}
Sep 26 16:38:13 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:13 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:13.716028 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:13.706475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:54381)
0926 16:38:13.707073 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:13.707088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:13.707194 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:13.707546 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:13.707551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:13.708014 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:13.708189 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:13.708198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.709636 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.709640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:13.710603 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:13.710615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.711526 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.711531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:13.711648 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:13.712264 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:13.712286 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:13.715669 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":302,"threads_started":1}
W20260926 16:38:13.716168 19920 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:13.706596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:54381 (local address 127.10.110.62:33509)
0926 16:38:13.706773 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:13.706779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:13.707153 (+ 374us) server_negotiation.cc:407] Connection header received
0926 16:38:13.707289 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:13.707294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:13.707394 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:13.707491 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:13.708328 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.709489 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.710731 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:13.711382 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:13.711500 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:13.715903 (+ 4403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:54381: BlockingRecv error: recv got EOF from 127.10.110.3:54381 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 16:38:14 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:14 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:14.050345 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:14.041185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:39207)
0926 16:38:14.041762 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:14.041779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:14.041919 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:14.042283 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:14.042288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:14.042692 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:14.042889 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:14.042903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.044338 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.044342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:14.045310 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:14.045324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.046010 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.046014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:14.046120 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:14.046674 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:14.046691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:14.050090 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
W20260926 16:38:14.050573 19920 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:14.041292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:39207 (local address 127.10.110.62:33509)
0926 16:38:14.041489 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:14.041496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:14.041864 (+ 368us) server_negotiation.cc:407] Connection header received
0926 16:38:14.042050 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:14.042054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:14.042129 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:14.042216 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:14.043051 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.044172 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.045448 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.045877 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.045967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:14.050323 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:39207: BlockingRecv error: recv got EOF from 127.10.110.2:39207 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 16:38:14 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:14 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:14.655933 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:14.646914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:33303)
0926 16:38:14.647413 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:14.647431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:14.647562 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:14.647924 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:14.647928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:14.648303 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:14.648445 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:14.648453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.649575 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.649581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:14.650929 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:14.650942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.651589 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.651593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:14.651708 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:14.652304 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:14.652321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:14.655636 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":249,"threads_started":1}
W20260926 16:38:14.656108 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:14.646972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:33303 (local address 127.10.110.62:33509)
0926 16:38:14.647479 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:14.647485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:14.647502 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:38:14.647661 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:14.647665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:14.647721 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:14.647808 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:14.648581 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.649451 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.651046 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.651478 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.651568 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:14.655870 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:33303: BlockingRecv error: recv got EOF from 127.10.110.1:33303 (error 108)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":245,"threads_started":1}
Sep 26 16:38:14 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:14 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:14.725961 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:14.716985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:56579)
0926 16:38:14.717451 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:14.717467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:14.717575 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:14.717840 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:14.717843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:14.718207 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:14.718406 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:14.718415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.719455 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.719461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:14.720887 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:14.720904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.721677 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.721684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:14.721861 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:14.722437 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:14.722455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:14.725604 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":221,"threads_started":1}
W20260926 16:38:14.726008 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:14.717106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:56579 (local address 127.10.110.62:33509)
0926 16:38:14.717292 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:14.717296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:14.717531 (+ 235us) server_negotiation.cc:407] Connection header received
0926 16:38:14.717657 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:14.717661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:14.717720 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:14.717807 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:14.718510 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.719288 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.721040 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:14.721489 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:14.721604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:14.725809 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:56579: BlockingRecv error: recv got EOF from 127.10.110.3:56579 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 16:38:15 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:15 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:15.060026 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:15.051311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:60151)
0926 16:38:15.051938 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:15.051954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:15.052090 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:15.052409 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:15.052414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:15.052804 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:15.052946 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:15.052954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.053922 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.053926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:15.054881 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:15.054892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.055541 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.055546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:15.055708 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:15.056273 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:15.056291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:15.059676 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":380,"threads_started":1}
W20260926 16:38:15.060165 19925 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:15.051483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:60151 (local address 127.10.110.62:33509)
0926 16:38:15.051692 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:15.051698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:15.052036 (+ 338us) server_negotiation.cc:407] Connection header received
0926 16:38:15.052201 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:15.052206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:15.052283 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:15.052364 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:15.053082 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.053798 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.055004 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.055411 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.055513 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:15.059911 (+ 4398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:60151: BlockingRecv error: recv got EOF from 127.10.110.2:60151 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 16:38:15 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:15 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:15.665383 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:15.656800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:42133)
0926 16:38:15.657323 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:15.657339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:15.657446 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:15.657734 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:15.657737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:15.658070 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:15.658269 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:15.658314 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.659319 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.659323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:15.660314 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:15.660325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.660990 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.660995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:15.661108 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:15.661661 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:15.661678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:15.665015 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":305,"threads_started":1}
W20260926 16:38:15.665485 19929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:15.656885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:42133 (local address 127.10.110.62:33509)
0926 16:38:15.657332 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:15.657336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:15.657399 (+ 63us) server_negotiation.cc:407] Connection header received
0926 16:38:15.657537 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:15.657541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:15.657611 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:15.657698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:15.658451 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.659163 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.660436 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.660848 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.660946 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:15.665251 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:42133: BlockingRecv error: recv got EOF from 127.10.110.1:42133 (error 108)
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
Sep 26 16:38:15 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:15 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:15.736012 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:15.726938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:37615)
0926 16:38:15.727450 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:15.727465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:15.727567 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:15.727938 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:15.727942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:15.728309 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:15.728453 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:15.728461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.729394 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.729398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:15.730337 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:15.730350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.730934 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.730937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:15.731055 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:15.731765 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:15.731786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:15.735676 (+ 3890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":302,"threads_started":1}
W20260926 16:38:15.736249 19929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:15.727039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:37615 (local address 127.10.110.62:33509)
0926 16:38:15.727195 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:15.727199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:15.727526 (+ 327us) server_negotiation.cc:407] Connection header received
0926 16:38:15.727659 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:15.727662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:15.727720 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:15.727795 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:15.728570 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.729275 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.730443 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:15.730827 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:15.730911 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:15.735909 (+ 4998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:37615: BlockingRecv error: recv got EOF from 127.10.110.3:37615 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 16:38:16 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:16 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:16.069224 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:16.060921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:46299)
0926 16:38:16.061396 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:16.061413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:16.061541 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:16.061855 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:16.061859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:16.062234 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:16.062386 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:16.062395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.063371 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.063375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:16.064382 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:16.064394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.065040 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.065045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:16.065187 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:16.065723 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:16.065740 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:16.068902 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
W20260926 16:38:16.069351 19929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:16.061036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:46299 (local address 127.10.110.62:33509)
0926 16:38:16.061216 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:16.061222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:16.061490 (+ 268us) server_negotiation.cc:407] Connection header received
0926 16:38:16.061656 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:16.061660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:16.061731 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:16.061806 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:16.062524 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.063230 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.064511 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.064903 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.065006 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:16.069085 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:46299: BlockingRecv error: recv got EOF from 127.10.110.2:46299 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 16:38:16 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:16 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:16.675040 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:16.666407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:39349)
0926 16:38:16.666881 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:16.666897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:16.666999 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:16.667287 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:16.667291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:16.667687 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:16.667892 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:16.667902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.669083 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.669087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:16.670113 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:16.670125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.670810 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.670814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:16.670928 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:16.671492 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:16.671510 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:16.674685 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
W20260926 16:38:16.675213 19933 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:16.666563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:39349 (local address 127.10.110.62:33509)
0926 16:38:16.666969 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:16.666973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:16.667006 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:38:16.667105 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:16.667109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:16.667171 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:16.667256 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:16.668026 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.668963 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.670236 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.670691 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.670781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:16.674937 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:39349: BlockingRecv error: recv got EOF from 127.10.110.1:39349 (error 108)
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":215,"threads_started":1}
Sep 26 16:38:16 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:16 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:16.745872 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:16.736887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:50811)
0926 16:38:16.737389 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:16.737404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:16.737526 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:16.737836 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:16.737841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:16.738194 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:16.738362 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:16.738371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.739378 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.739382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:16.740780 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:16.740792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.741503 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.741508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:16.741619 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:16.742231 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:16.742256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:16.745617 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":241,"threads_started":1}
W20260926 16:38:16.746047 19933 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:16.737013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:50811 (local address 127.10.110.62:33509)
0926 16:38:16.737222 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:16.737228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:16.737479 (+ 251us) server_negotiation.cc:407] Connection header received
0926 16:38:16.737635 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:16.737640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:16.737702 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:16.737777 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:16.738502 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.739230 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.740938 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:16.741367 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:16.741468 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:16.745833 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:50811: BlockingRecv error: recv got EOF from 127.10.110.3:50811 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 16:38:17 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:17 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:17.082142 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:17.070197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:47739)
0926 16:38:17.070664 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:17.070680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:17.070851 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:17.071152 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:17.071156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:17.071542 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:17.071735 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:17.071745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.072735 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.072738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:17.073652 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:17.073664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.074301 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.074306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:17.074424 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:17.078533 (+ 4109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:17.078552 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:17.081814 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
W20260926 16:38:17.082307 19933 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:17.070289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:47739 (local address 127.10.110.62:33509)
0926 16:38:17.070483 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:17.070488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:17.070773 (+ 285us) server_negotiation.cc:407] Connection header received
0926 16:38:17.070955 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:17.070959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:17.071021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:17.071101 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:17.071895 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.072619 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.073782 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.074170 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.074257 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:17.082077 (+ 7820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:47739: BlockingRecv error: recv got EOF from 127.10.110.2:47739 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 16:38:17 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:17 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:17.684935 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:17.676153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:44919)
0926 16:38:17.676743 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:17.676763 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:17.676892 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:17.677408 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:17.677413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:17.677837 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:17.678069 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:17.678085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.679187 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.679192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:17.680199 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:17.680212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.680932 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.680937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:17.681074 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:17.681681 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:17.681699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:17.684699 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":292,"threads_started":1}
W20260926 16:38:17.685105 19937 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:17.676249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:44919 (local address 127.10.110.62:33509)
0926 16:38:17.677057 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:17.677063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:17.677079 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:38:17.677126 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:17.677130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:17.677208 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:17.677314 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:17.678269 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.679043 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.680327 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.680743 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.680843 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:17.684882 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:44919: BlockingRecv error: recv got EOF from 127.10.110.1:44919 (error 108)
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":462,"threads_started":1}
Sep 26 16:38:17 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:17 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:17.755123 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:17.746728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:44113)
0926 16:38:17.747207 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:17.747222 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:17.747330 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:17.747613 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:17.747616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:17.748149 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:17.748360 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:17.748373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.749450 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.749455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:17.750441 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:17.750453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.751073 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.751077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:17.751190 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:17.751690 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:17.751706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:17.754916 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":274,"threads_started":1}
W20260926 16:38:17.755286 19937 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:17.746837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:44113 (local address 127.10.110.62:33509)
0926 16:38:17.747001 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:17.747006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:17.747318 (+ 312us) server_negotiation.cc:407] Connection header received
0926 16:38:17.747422 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:17.747426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:17.747491 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:17.747576 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:17.748534 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.749316 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.750571 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:17.750963 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:17.751052 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:17.755097 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:44113: BlockingRecv error: recv got EOF from 127.10.110.3:44113 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 16:38:18 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:18 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:18.092039 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:18.083257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:49959)
0926 16:38:18.083821 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:18.083873 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:18.083994 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:18.084380 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:18.084383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:18.084871 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:18.085047 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:18.085056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.086043 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.086046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:18.086943 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:18.086953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.087707 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.087712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:18.087896 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:18.088495 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:18.088513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:18.091737 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
W20260926 16:38:18.092213 19937 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:18.083310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:49959 (local address 127.10.110.62:33509)
0926 16:38:18.083475 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:18.083481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:18.083950 (+ 469us) server_negotiation.cc:407] Connection header received
0926 16:38:18.084110 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:18.084115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:18.084199 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:18.084320 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:18.085167 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.085926 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.087081 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.087527 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.087626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:18.091946 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:49959: BlockingRecv error: recv got EOF from 127.10.110.2:49959 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 16:38:18 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:18 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:18.694222 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:18.685766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:56631)
0926 16:38:18.686219 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:18.686235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:18.686341 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:18.686714 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:18.686717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:18.687088 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:18.687230 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:18.687238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.688234 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.688238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:18.689147 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:18.689160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.689934 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.689938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:18.690090 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:18.690610 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:18.690627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:18.693993 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
W20260926 16:38:18.694459 19941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:18.685854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:56631 (local address 127.10.110.62:33509)
0926 16:38:18.686476 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:18.686480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:18.686494 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:38:18.686546 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:18.686549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:18.686603 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:18.686685 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:18.687363 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.688104 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.689297 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.689774 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.689877 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:18.694202 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:56631: BlockingRecv error: recv got EOF from 127.10.110.1:56631 (error 108)
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":465,"threads_started":1}
Sep 26 16:38:18 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:18 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:18.764436 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:18.755998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:47099)
0926 16:38:18.756485 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:18.756500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:18.756612 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:18.756866 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:18.756869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:18.757239 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:18.757429 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:18.757438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.758467 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.758472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:18.759446 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:18.759459 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.760186 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.760191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:18.760325 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:18.760836 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:18.760854 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:18.764131 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":251,"threads_started":1}
W20260926 16:38:18.764536 19941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:18.756026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:47099 (local address 127.10.110.62:33509)
0926 16:38:18.756171 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:18.756175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:18.756565 (+ 390us) server_negotiation.cc:407] Connection header received
0926 16:38:18.756688 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:18.756692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:18.756749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:18.756830 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:18.757572 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.758318 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.759577 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:18.760017 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:18.760106 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:18.764336 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:47099: BlockingRecv error: recv got EOF from 127.10.110.3:47099 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 16:38:19 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:19 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:19.101250 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:19.092981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:54589)
0926 16:38:19.093427 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:19.093443 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:19.093556 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:19.093886 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:19.093890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:19.094257 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:19.094412 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:19.094422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.095418 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.095421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:19.096506 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:19.096518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.097330 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.097334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:19.097478 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:19.097983 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:19.097999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:19.101028 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
W20260926 16:38:19.101450 19941 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:19.093156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:54589 (local address 127.10.110.62:33509)
0926 16:38:19.093349 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:19.093356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:19.093510 (+ 154us) server_negotiation.cc:407] Connection header received
0926 16:38:19.093644 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:19.093648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:19.093727 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:19.093807 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:19.094527 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.095274 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.096648 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.097215 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.097304 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:19.101207 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:54589: BlockingRecv error: recv got EOF from 127.10.110.2:54589 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 16:38:19 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:19 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.1@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:19.704058 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:19.695292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.1:49677)
0926 16:38:19.695772 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:19.695789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:19.695925 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:19.696190 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:19.696193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:19.696530 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:19.696707 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:19.696715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.697636 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.697639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:19.698561 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:19.698572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.699378 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.699383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:19.699516 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:19.700170 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:19.700191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:19.703750 (+ 3559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
W20260926 16:38:19.704118 19945 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:19.695406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:49677 (local address 127.10.110.62:33509)
0926 16:38:19.695858 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:19.695862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:19.695878 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:38:19.696022 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:19.696025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:19.696081 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:19.696160 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:19.696833 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.697532 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.698698 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.699237 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.699332 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:19.703942 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.1:49677: BlockingRecv error: recv got EOF from 127.10.110.1:49677 (error 108)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":213,"threads_started":1}
Sep 26 16:38:19 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:19 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.3@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:19.774050 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:19.765403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.3:42111)
0926 16:38:19.765830 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:19.765845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:19.765952 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:19.766193 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:19.766196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:19.766596 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:19.766735 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:19.766744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.767693 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.767696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:19.768682 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:19.768693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.769359 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.769363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:19.769492 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:19.770015 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:19.770031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:19.773810 (+ 3779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
W20260926 16:38:19.774161 19945 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:19.765517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:42111 (local address 127.10.110.62:33509)
0926 16:38:19.765740 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:19.765745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:19.765906 (+ 161us) server_negotiation.cc:407] Connection header received
0926 16:38:19.766027 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:19.766030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:19.766088 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:19.766165 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:19.766841 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.767584 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.768837 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:19.769240 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:19.769338 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:19.773981 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.3:42111: BlockingRecv error: recv got EOF from 127.10.110.3:42111 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 26 16:38:20 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:38:20 dist-test-slave-whjp krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.110.2@KRBTEST.COM for oryx/127.10.110.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:38:20.110643 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:20.102330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33509 (local address 127.10.110.2:34733)
0926 16:38:20.102787 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:20.102802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:20.102920 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:20.103243 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:20.103248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:20.103606 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:20.103777 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:20.103787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.104748 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.104752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:20.105681 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:20.105695 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.106341 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.106347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:20.106505 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:20.107079 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:20.107095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:20.110388 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.110.62:33509: Server oryx/127.10.110.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
W20260926 16:38:20.110813 19945 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:20.102421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:34733 (local address 127.10.110.62:33509)
0926 16:38:20.102585 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:20.102590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:20.102881 (+ 291us) server_negotiation.cc:407] Connection header received
0926 16:38:20.103028 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:20.103034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:20.103108 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:20.103185 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:20.103924 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.104623 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.105810 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.106208 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.106311 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:20.110574 (+ 4263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.110.2:34733: BlockingRecv error: recv got EOF from 127.10.110.2:34733 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:38:20.619079 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19474
I20260926 16:38:20.630985 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:20.620131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.10.110.1:42585)
0926 16:38:20.620711 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:20.620718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:20.620763 (+ 45us) server_negotiation.cc:407] Connection header received
0926 16:38:20.620846 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:20.620856 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:20.621131 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:20.621326 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:20.622724 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.623872 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.625051 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.625577 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.625675 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:20.628031 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:20.628040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:20.628054 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:20.628103 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:20.630489 (+ 2386us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.110.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:38:20.630762 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58118: Request ticket server kudu/127.10.110.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":281,"threads_started":1}
W20260926 16:38:20.631090 19949 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58118: Request ticket server kudu/127.10.110.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:38:20.631089 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19474: Not authorized: Client connection negotiation failed: client connection to 127.10.110.1:42585: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.110.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:38:20.631290 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19474
I20260926 16:38:20.644974 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19625
I20260926 16:38:20.658200 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:20.645986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.10.110.2:35755)
0926 16:38:20.648277 (+ 2291us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:20.648284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:20.648311 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:38:20.648377 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:20.648383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:20.648605 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:20.648792 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:20.650024 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.651062 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.652255 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.652793 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.652904 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:20.655158 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:20.655170 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:20.655192 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:20.655249 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:20.657591 (+ 2342us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.110.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:38:20.657898 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53040: Request ticket server kudu/127.10.110.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":2134,"thread_start_us":2019,"threads_started":1}
W20260926 16:38:20.658303 19950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53040: Request ticket server kudu/127.10.110.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:38:20.658285 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19625: Not authorized: Client connection negotiation failed: client connection to 127.10.110.2:35755: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.110.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:38:20.658447 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19625
I20260926 16:38:20.671339 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19757
I20260926 16:38:20.683120 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:20.672463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.10.110.3:37493)
0926 16:38:20.673006 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:20.673013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:20.673038 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:38:20.673101 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:20.673108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:20.673329 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:20.673505 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:20.674789 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.675946 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.677131 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:20.677673 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:20.677792 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:20.680142 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:20.680152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:20.680167 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:20.680203 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:20.682460 (+ 2257us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.110.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:38:20.682787 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47722: Request ticket server kudu/127.10.110.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
W20260926 16:38:20.683148 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19757: Not authorized: Client connection negotiation failed: client connection to 127.10.110.3:37493: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.10.110.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:38:20.683208 19951 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47722: Request ticket server kudu/127.10.110.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:38:20.683296 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19757
I20260926 16:38:20.696229 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19405
I20260926 16:38:20.697374 19467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:20.808552 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19405
2026-09-26T16:38:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21080 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T16:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:38:20Z Disabled control of system clock
I20260926 16:38:20.874668 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:41595
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:41595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:38:21.192818 19956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:21.193099 19956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:21.193212 19956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:21.200282 19956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:38:21.200397 19956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:21.200435 19956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:21.200490 19956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:38:21.200615 19956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:38:21.209842 19956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:41595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:41595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:21.211179 19956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:21.212989 19956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:21.224462 19961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:21.224501 19962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:21.224668 19964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:21.225164 19956 server_base.cc:1034] running on GCE node
I20260926 16:38:21.225891 19956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:21.227218 19956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:21.228411 19956 hybrid_clock.cc:648] HybridClock initialized: now 1790440701228348 us; error 53 us; skew 500 ppm
I20260926 16:38:21.228753 19956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:21.230899 19956 webserver.cc:460] Webserver started at http://127.10.110.62:34939/ using document root <none> and password file <none>
I20260926 16:38:21.231602 19956 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:21.231693 19956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:21.232034 19956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:21.234843 19956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "5b144fcf1db44c13a6e4aad268baca7d"
format_stamp: "Formatted at 2026-09-26 16:38:21 on dist-test-slave-whjp"
server_key: "859c14c627cffdb6f354df95a419ca97"
server_key_iv: "5b5f5c65aa04d7716803385d7e5947c4"
server_key_version: "encryptionkey@0"
I20260926 16:38:21.235575 19956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "5b144fcf1db44c13a6e4aad268baca7d"
format_stamp: "Formatted at 2026-09-26 16:38:21 on dist-test-slave-whjp"
server_key: "859c14c627cffdb6f354df95a419ca97"
server_key_iv: "5b5f5c65aa04d7716803385d7e5947c4"
server_key_version: "encryptionkey@0"
I20260926 16:38:21.240859 19956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:38:21.244375 19970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:21.245740 19956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:38:21.245937 19956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "5b144fcf1db44c13a6e4aad268baca7d"
format_stamp: "Formatted at 2026-09-26 16:38:21 on dist-test-slave-whjp"
server_key: "859c14c627cffdb6f354df95a419ca97"
server_key_iv: "5b5f5c65aa04d7716803385d7e5947c4"
server_key_version: "encryptionkey@0"
I20260926 16:38:21.246151 19956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:21.264817 19956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:21.265645 19956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:21.265905 19956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:21.282151 19956 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:41595
I20260926 16:38:21.282128 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:41595 every 8 connection(s)
I20260926 16:38:21.283465 19956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:38:21.287215 20022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:38:21.292537 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 19956
I20260926 16:38:21.292707 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:38:21.293390 10680 external_mini_cluster.cc:1428] Setting key afb63eec0de5d79cd97ef5bf8e33e0bd
I20260926 16:38:21.298566 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d: Bootstrap starting.
I20260926 16:38:21.302512 20022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:21.303577 20022 log.cc:824] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:21.306725 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d: No bootstrap required, opened a new log
I20260926 16:38:21.309432 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:21.295718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.10.110.62:41595)
0926 16:38:21.296829 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:21.296844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:21.296895 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:38:21.299194 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:21.299225 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:21.299854 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:21.300327 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:21.300899 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:21.302735 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:21.304256 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:21.305046 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:21.305225 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:21.306533 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:21.306544 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:21.306562 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:21.307289 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:21.307401 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:21.307461 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:21.307579 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:21.307783 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 16:38:21.308142 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":313,"threads_started":1}
I20260926 16:38:21.314898 20022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b144fcf1db44c13a6e4aad268baca7d" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41595 } }
I20260926 16:38:21.315220 20022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:21.315294 20022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b144fcf1db44c13a6e4aad268baca7d, State: Initialized, Role: FOLLOWER
I20260926 16:38:21.315771 20022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b144fcf1db44c13a6e4aad268baca7d" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41595 } }
I20260926 16:38:21.315997 20022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:38:21.316094 20022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:38:21.316206 20022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:21.318538 20022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b144fcf1db44c13a6e4aad268baca7d" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41595 } }
I20260926 16:38:21.318984 20022 leader_election.cc:302] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b144fcf1db44c13a6e4aad268baca7d; no voters:
I20260926 16:38:21.319458 20022 leader_election.cc:288] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:38:21.319599 20027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:38:21.320235 20027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [term 1 LEADER]: Becoming Leader. State: Replica: 5b144fcf1db44c13a6e4aad268baca7d, State: Running, Role: LEADER
I20260926 16:38:21.320787 20027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b144fcf1db44c13a6e4aad268baca7d" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41595 } }
I20260926 16:38:21.321525 20022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:38:21.323329 20028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b144fcf1db44c13a6e4aad268baca7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b144fcf1db44c13a6e4aad268baca7d" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41595 } } }
I20260926 16:38:21.323719 20028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:21.324167 20029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b144fcf1db44c13a6e4aad268baca7d. Latest consensus state: current_term: 1 leader_uuid: "5b144fcf1db44c13a6e4aad268baca7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b144fcf1db44c13a6e4aad268baca7d" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41595 } } }
I20260926 16:38:21.324536 20029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:21.325213 20036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:38:21.330102 20036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:38:21.338281 20036 catalog_manager.cc:1347] Generated new cluster ID: 090daf9bfbf24c97bd36fed54cc1750c
I20260926 16:38:21.338378 20036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:38:21.353176 20036 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:38:21.354451 20036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:38:21.368163 20036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b144fcf1db44c13a6e4aad268baca7d: Generated new TSK 0
I20260926 16:38:21.368974 20036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:38:21.383193 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41595
--builtin_ntp_servers=127.10.110.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 16:38:21.720818 20046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:21.721086 20046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:21.721206 20046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:21.728453 20046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:21.728547 20046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:21.728670 20046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:38:21.737828 20046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:21.739033 20046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:21.740739 20046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:21.753345 20051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:21.753517 20054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:21.753476 20052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:21.754463 20046 server_base.cc:1034] running on GCE node
I20260926 16:38:21.755039 20046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:21.755937 20046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:21.757146 20046 hybrid_clock.cc:648] HybridClock initialized: now 1790440701757089 us; error 48 us; skew 500 ppm
I20260926 16:38:21.757444 20046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:21.759521 20046 webserver.cc:460] Webserver started at http://127.10.110.1:38503/ using document root <none> and password file <none>
I20260926 16:38:21.760213 20046 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:21.760310 20046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:21.760596 20046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:21.763307 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "7a8406c7cc5444ea9152fa7060132467"
format_stamp: "Formatted at 2026-09-26 16:38:21 on dist-test-slave-whjp"
server_key: "34ee9135a16edaa0d3d6a81dc63eeea3"
server_key_iv: "24a1554297340de472130e6e9357a362"
server_key_version: "encryptionkey@0"
I20260926 16:38:21.764099 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "7a8406c7cc5444ea9152fa7060132467"
format_stamp: "Formatted at 2026-09-26 16:38:21 on dist-test-slave-whjp"
server_key: "34ee9135a16edaa0d3d6a81dc63eeea3"
server_key_iv: "24a1554297340de472130e6e9357a362"
server_key_version: "encryptionkey@0"
I20260926 16:38:21.769064 20046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:38:21.772352 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:21.773618 20046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:38:21.773777 20046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "7a8406c7cc5444ea9152fa7060132467"
format_stamp: "Formatted at 2026-09-26 16:38:21 on dist-test-slave-whjp"
server_key: "34ee9135a16edaa0d3d6a81dc63eeea3"
server_key_iv: "24a1554297340de472130e6e9357a362"
server_key_version: "encryptionkey@0"
I20260926 16:38:21.773985 20046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:21.801959 20046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:21.802762 20046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:21.803050 20046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:21.804028 20046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:21.805415 20046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:21.805507 20046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:21.805619 20046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:21.805691 20046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:21.834645 20046 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:33911
I20260926 16:38:21.834688 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:33911 every 8 connection(s)
I20260926 16:38:21.835809 20046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:38:21.836460 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20046
I20260926 16:38:21.836653 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:38:21.837327 10680 external_mini_cluster.cc:1428] Setting key 1ec4bb1f8b44f08af9fc8237ec14c489
I20260926 16:38:21.840174 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41595
--builtin_ntp_servers=127.10.110.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:38:21.849335 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:21.838357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:50833 (local address 127.10.110.62:41595)
0926 16:38:21.838734 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:21.838739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:21.840049 (+ 1310us) server_negotiation.cc:407] Connection header received
0926 16:38:21.842745 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:21.842751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:21.842839 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:21.842930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:21.844014 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:21.844743 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:21.846650 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:21.847224 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:21.847350 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:21.848575 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:21.848581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:21.848586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:21.848606 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:21.848648 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:21.848660 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:21.848735 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:21.848912 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 16:38:21.848997 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":203,"threads_started":1}
I20260926 16:38:21.850416 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:21.839034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41595 (local address 127.10.110.1:50833)
0926 16:38:21.839922 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:21.839957 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:21.842495 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:21.843168 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:21.843177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:21.843268 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:21.843818 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:21.843865 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:21.844890 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:21.844895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:21.846498 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:21.846514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:21.847421 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:21.847425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:21.847621 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:21.848255 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:38:21.848259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:21.848325 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:38:21.848416 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:21.848742 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:21.848748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:21.848753 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:38:21.848880 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 16:38:21.849191 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":282,"threads_started":1}
I20260926 16:38:21.851934 20173 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41595
I20260926 16:38:21.852203 20173 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:21.852905 20173 heartbeater.cc:511] Master 127.10.110.62:41595 requested a full tablet report, sending...
I20260926 16:38:21.855100 19987 ts_manager.cc:194] Registered new tserver with Master: 7a8406c7cc5444ea9152fa7060132467 (127.10.110.1:33911)
I20260926 16:38:21.857167 19987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.1:50833
W20260926 16:38:22.148370 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:22.148635 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:22.148723 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:22.155893 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:22.155987 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:22.156095 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:38:22.164839 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:22.165974 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:22.167623 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:22.179672 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:22.179736 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:22.179711 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:22.180549 20177 server_base.cc:1034] running on GCE node
I20260926 16:38:22.181121 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:22.182013 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:22.183192 20177 hybrid_clock.cc:648] HybridClock initialized: now 1790440702183152 us; error 33 us; skew 500 ppm
I20260926 16:38:22.183485 20177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:22.191114 20177 webserver.cc:460] Webserver started at http://127.10.110.2:35455/ using document root <none> and password file <none>
I20260926 16:38:22.191867 20177 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:22.191973 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:22.192277 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:22.195063 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "8e8e5fa5ee834a20a59173d8431724de"
format_stamp: "Formatted at 2026-09-26 16:38:22 on dist-test-slave-whjp"
server_key: "18f887f8cb34138f75cc40d13e8d0ce9"
server_key_iv: "5d9ef114d7419cc27ad9f688c167da9b"
server_key_version: "encryptionkey@0"
I20260926 16:38:22.195804 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "8e8e5fa5ee834a20a59173d8431724de"
format_stamp: "Formatted at 2026-09-26 16:38:22 on dist-test-slave-whjp"
server_key: "18f887f8cb34138f75cc40d13e8d0ce9"
server_key_iv: "5d9ef114d7419cc27ad9f688c167da9b"
server_key_version: "encryptionkey@0"
I20260926 16:38:22.200745 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:38:22.204057 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:22.205408 20177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 16:38:22.205569 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "8e8e5fa5ee834a20a59173d8431724de"
format_stamp: "Formatted at 2026-09-26 16:38:22 on dist-test-slave-whjp"
server_key: "18f887f8cb34138f75cc40d13e8d0ce9"
server_key_iv: "5d9ef114d7419cc27ad9f688c167da9b"
server_key_version: "encryptionkey@0"
I20260926 16:38:22.205793 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:22.230938 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:22.231864 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:22.232190 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:22.233156 20177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:22.234684 20177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:22.234786 20177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:22.234900 20177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:22.234977 20177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:22.268090 20177 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:32867
I20260926 16:38:22.268157 20303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:32867 every 8 connection(s)
I20260926 16:38:22.269378 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:38:22.271273 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20177
I20260926 16:38:22.271420 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:38:22.272001 10680 external_mini_cluster.cc:1428] Setting key 32d2add2e11e39a55fe66afb14a726c3
I20260926 16:38:22.274485 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41595
--builtin_ntp_servers=127.10.110.20:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:38:22.282137 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.272013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:36699 (local address 127.10.110.62:41595)
0926 16:38:22.272196 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:22.272201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:22.273850 (+ 1649us) server_negotiation.cc:407] Connection header received
0926 16:38:22.275668 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:22.275674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:22.275745 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:22.275865 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:22.276927 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.277891 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.279445 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.280001 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.280115 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.281389 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:22.281394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:22.281398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:22.281415 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:22.281445 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:22.281457 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:22.281528 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:22.281752 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 16:38:22.281847 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:38:22.283278 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.272578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41595 (local address 127.10.110.2:36699)
0926 16:38:22.273689 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:22.273741 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:22.275411 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:22.276084 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:22.276092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:22.276180 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:22.276759 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:22.276783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.278019 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.278022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:22.279313 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:22.279326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.280227 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.280232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:22.280438 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.281072 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:38:22.281074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:22.281110 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:38:22.281186 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:22.281542 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:22.281553 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:22.281560 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:38:22.281729 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 16:38:22.282098 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":337,"threads_started":1}
I20260926 16:38:22.284654 20304 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41595
I20260926 16:38:22.284943 20304 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:22.285732 20304 heartbeater.cc:511] Master 127.10.110.62:41595 requested a full tablet report, sending...
I20260926 16:38:22.287215 19987 ts_manager.cc:194] Registered new tserver with Master: 8e8e5fa5ee834a20a59173d8431724de (127.10.110.2:32867)
I20260926 16:38:22.288266 19987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.2:36699
W20260926 16:38:22.590221 20307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:22.590502 20307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:22.590608 20307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:22.597832 20307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:22.597931 20307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:22.598062 20307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:38:22.607132 20307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:22.608424 20307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:22.610298 20307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:22.622753 20313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:22.622831 20315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:22.623257 20312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:22.624008 20307 server_base.cc:1034] running on GCE node
I20260926 16:38:22.624641 20307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:22.625592 20307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:22.626788 20307 hybrid_clock.cc:648] HybridClock initialized: now 1790440702626722 us; error 60 us; skew 500 ppm
I20260926 16:38:22.627096 20307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:22.629544 20307 webserver.cc:460] Webserver started at http://127.10.110.3:37405/ using document root <none> and password file <none>
I20260926 16:38:22.630223 20307 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:22.630311 20307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:22.630600 20307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:22.633513 20307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "4a19fb5d3d194dbf9753d67376453079"
format_stamp: "Formatted at 2026-09-26 16:38:22 on dist-test-slave-whjp"
server_key: "c87686de51644bfc076344ee66268b44"
server_key_iv: "fce4c3aa96b36ee281865e075ba938cf"
server_key_version: "encryptionkey@0"
I20260926 16:38:22.634245 20307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "4a19fb5d3d194dbf9753d67376453079"
format_stamp: "Formatted at 2026-09-26 16:38:22 on dist-test-slave-whjp"
server_key: "c87686de51644bfc076344ee66268b44"
server_key_iv: "fce4c3aa96b36ee281865e075ba938cf"
server_key_version: "encryptionkey@0"
I20260926 16:38:22.639148 20307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 16:38:22.642470 20321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:22.643709 20307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:38:22.643911 20307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "4a19fb5d3d194dbf9753d67376453079"
format_stamp: "Formatted at 2026-09-26 16:38:22 on dist-test-slave-whjp"
server_key: "c87686de51644bfc076344ee66268b44"
server_key_iv: "fce4c3aa96b36ee281865e075ba938cf"
server_key_version: "encryptionkey@0"
I20260926 16:38:22.644102 20307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:22.672940 20307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:22.673749 20307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:22.674036 20307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:22.674952 20307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:22.676340 20307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:22.676432 20307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:22.676535 20307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:22.676601 20307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:22.707259 20307 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:42501
I20260926 16:38:22.707329 20433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:42501 every 8 connection(s)
I20260926 16:38:22.708822 20307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:38:22.717519 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20307
I20260926 16:38:22.717665 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:38:22.718072 10680 external_mini_cluster.cc:1428] Setting key e25cacf47b4e61d62d496ec44c0ca16e
I20260926 16:38:22.721377 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.711562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:48845 (local address 127.10.110.62:41595)
0926 16:38:22.711757 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:22.711763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:22.713237 (+ 1474us) server_negotiation.cc:407] Connection header received
0926 16:38:22.715029 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:22.715034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:22.715101 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:22.715189 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:22.716284 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.716996 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.718614 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.719091 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.719214 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.720560 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:22.720566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:22.720571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:22.720592 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:22.720630 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:22.720645 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:22.720731 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:22.720954 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 16:38:22.721077 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:38:22.722366 20436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.712176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41595 (local address 127.10.110.3:48845)
0926 16:38:22.713084 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:22.713120 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:22.714778 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:22.715429 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:22.715438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:22.715527 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:22.716133 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:22.716150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.717133 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.717137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:22.718473 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:22.718487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.719305 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.719312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:22.719553 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.720247 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:38:22.720250 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:22.720290 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:38:22.720377 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:22.720745 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:22.720756 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:22.720764 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 16:38:22.720909 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 16:38:22.721321 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":279,"threads_started":1}
I20260926 16:38:22.723603 20434 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41595
I20260926 16:38:22.723918 20434 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:22.724668 20434 heartbeater.cc:511] Master 127.10.110.62:41595 requested a full tablet report, sending...
I20260926 16:38:22.725922 19987 ts_manager.cc:194] Registered new tserver with Master: 4a19fb5d3d194dbf9753d67376453079 (127.10.110.3:42501)
I20260926 16:38:22.726691 19987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.3:48845
I20260926 16:38:22.733613 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:38:22.739283 20175 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:22.737887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.10.110.62:41595)
0926 16:38:22.738031 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:22.738035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:22.738480 (+ 445us) server_negotiation.cc:407] Connection header received
0926 16:38:22.738619 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:22.738623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:22.738683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:22.738766 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:22.739104 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43204: BlockingRecv error: recv got EOF from 127.0.0.1:43204 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:38:22.739737 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20046
I20260926 16:38:22.747005 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.740671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.10.110.1:33911)
0926 16:38:22.741189 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:22.741195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:22.741220 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:38:22.741280 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:22.741285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:22.741488 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:22.741606 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:22.742006 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.742960 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.744177 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.744697 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.744787 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.745685 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:22.745692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:22.745704 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:22.746282 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:22.746325 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:22.746344 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:22.746436 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:22.746551 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 16:38:22.746654 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
I20260926 16:38:22.748253 20168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:22.831431 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20046
I20260926 16:38:22.847206 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20177
I20260926 16:38:22.854633 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.848134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.10.110.2:32867)
0926 16:38:22.848666 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:22.848673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:22.848701 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:38:22.848760 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:22.848766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:22.848980 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:22.849109 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:22.849528 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.850624 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.851883 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.852426 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.852536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.853384 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:22.853391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:22.853405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:22.853990 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:22.854036 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:22.854055 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:22.854139 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:22.854240 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 16:38:22.854344 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":251,"threads_started":1}
I20260926 16:38:22.855824 20299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:22.935472 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20177
I20260926 16:38:22.948536 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20307
I20260926 16:38:22.956302 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:22.949395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.10.110.3:42501)
0926 16:38:22.950061 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:22.950068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:22.950098 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:38:22.950178 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:22.950184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:22.950399 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:22.950542 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:22.950988 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.952033 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.953225 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:22.953769 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:22.953896 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:22.954895 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:22.954903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:22.954918 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:22.955476 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:22.955522 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:22.955551 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:22.955648 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:22.955803 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 16:38:22.955970 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":358,"threads_started":1}
I20260926 16:38:22.957553 20429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:23.037901 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20307
I20260926 16:38:23.050154 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 19956
I20260926 16:38:23.051311 20017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:23.152786 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 19956
2026-09-26T16:38:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2339 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T16:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:38:23Z Disabled control of system clock
I20260926 16:38:23.217171 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:32883
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:38:23.545849 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:23.546130 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:23.546211 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:23.553390 20453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:38:23.553508 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:23.553544 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:23.553598 20453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:38:23.553640 20453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:38:23.565045 20453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:32883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:23.566697 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:23.568933 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:23.580461 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:23.580431 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:23.580472 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:23.581287 20453 server_base.cc:1034] running on GCE node
I20260926 16:38:23.582067 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:23.583467 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:23.584658 20453 hybrid_clock.cc:648] HybridClock initialized: now 1790440703584606 us; error 48 us; skew 500 ppm
I20260926 16:38:23.585013 20453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:23.587141 20453 webserver.cc:460] Webserver started at http://127.10.110.62:37539/ using document root <none> and password file <none>
I20260926 16:38:23.587806 20453 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:23.587962 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:23.588269 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:23.591095 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "07745f5aadfa495eb9332edfe378959e"
format_stamp: "Formatted at 2026-09-26 16:38:23 on dist-test-slave-whjp"
server_key: "cb68b99eb38ffe4ac2fc7ae4664c1620"
server_key_iv: "711a0d3fa41b0f5084afe4c713e9139c"
server_key_version: "encryptionkey@0"
I20260926 16:38:23.591900 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "07745f5aadfa495eb9332edfe378959e"
format_stamp: "Formatted at 2026-09-26 16:38:23 on dist-test-slave-whjp"
server_key: "cb68b99eb38ffe4ac2fc7ae4664c1620"
server_key_iv: "711a0d3fa41b0f5084afe4c713e9139c"
server_key_version: "encryptionkey@0"
I20260926 16:38:23.596830 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:38:23.600057 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:23.601315 20453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:38:23.601475 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "07745f5aadfa495eb9332edfe378959e"
format_stamp: "Formatted at 2026-09-26 16:38:23 on dist-test-slave-whjp"
server_key: "cb68b99eb38ffe4ac2fc7ae4664c1620"
server_key_iv: "711a0d3fa41b0f5084afe4c713e9139c"
server_key_version: "encryptionkey@0"
I20260926 16:38:23.601658 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:23.607131 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:23.607823 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:23.608150 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:23.625092 20453 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:32883
I20260926 16:38:23.625110 20518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:32883 every 8 connection(s)
I20260926 16:38:23.626422 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:38:23.627111 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20453
I20260926 16:38:23.627295 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:38:23.628050 10680 external_mini_cluster.cc:1428] Setting key e14293b499a5d460e8d650ce4c663c0a
I20260926 16:38:23.630602 20519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:38:23.637773 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:23.630199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.10.110.62:32883)
0926 16:38:23.631354 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:23.631369 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:23.631445 (+ 76us) server_negotiation.cc:407] Connection header received
0926 16:38:23.633578 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:23.633606 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:23.634177 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:23.634641 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:23.635685 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:23.635693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:23.635711 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:23.635727 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:23.635853 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:23.635885 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:23.636108 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:23.636240 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 16:38:23.636522 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":389,"threads_started":1}
I20260926 16:38:23.643754 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e: Bootstrap starting.
I20260926 16:38:23.646847 20519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:23.647796 20519 log.cc:824] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:23.650614 20519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e: No bootstrap required, opened a new log
I20260926 16:38:23.657691 20519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07745f5aadfa495eb9332edfe378959e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32883 } }
I20260926 16:38:23.658038 20519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:23.658108 20519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07745f5aadfa495eb9332edfe378959e, State: Initialized, Role: FOLLOWER
I20260926 16:38:23.658596 20519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07745f5aadfa495eb9332edfe378959e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32883 } }
I20260926 16:38:23.658787 20519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:38:23.658880 20519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:38:23.658998 20519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:23.661044 20519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07745f5aadfa495eb9332edfe378959e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32883 } }
I20260926 16:38:23.661502 20519 leader_election.cc:302] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07745f5aadfa495eb9332edfe378959e; no voters:
I20260926 16:38:23.662045 20519 leader_election.cc:288] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:38:23.662174 20524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:38:23.662891 20524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [term 1 LEADER]: Becoming Leader. State: Replica: 07745f5aadfa495eb9332edfe378959e, State: Running, Role: LEADER
I20260926 16:38:23.663408 20524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07745f5aadfa495eb9332edfe378959e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32883 } }
I20260926 16:38:23.664045 20519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:38:23.665938 20526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07745f5aadfa495eb9332edfe378959e. Latest consensus state: current_term: 1 leader_uuid: "07745f5aadfa495eb9332edfe378959e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07745f5aadfa495eb9332edfe378959e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32883 } } }
I20260926 16:38:23.666221 20526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:23.666977 20530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:38:23.666841 20525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07745f5aadfa495eb9332edfe378959e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07745f5aadfa495eb9332edfe378959e" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 32883 } } }
I20260926 16:38:23.667773 20525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:23.675534 20530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:38:23.685614 20530 catalog_manager.cc:1347] Generated new cluster ID: 0ae5367366214d718c3f60dcf080f2b2
I20260926 16:38:23.685732 20530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:38:23.710175 20530 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:38:23.710691 20530 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:38:23.718830 20530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07745f5aadfa495eb9332edfe378959e: Generated new TSK 0
I20260926 16:38:23.719965 20530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:38:23.733713 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32883
--builtin_ntp_servers=127.10.110.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:38:24.057147 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:24.057408 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:24.057520 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:24.064517 20543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:24.064600 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:24.064738 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:38:24.073676 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:24.074856 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:24.076639 20543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:24.089175 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:24.089238 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:24.089217 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:24.089987 20543 server_base.cc:1034] running on GCE node
I20260926 16:38:24.090663 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:24.091575 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:24.092756 20543 hybrid_clock.cc:648] HybridClock initialized: now 1790440704092710 us; error 41 us; skew 500 ppm
I20260926 16:38:24.093075 20543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:24.095180 20543 webserver.cc:460] Webserver started at http://127.10.110.1:40257/ using document root <none> and password file <none>
I20260926 16:38:24.095857 20543 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:24.095969 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:24.096257 20543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:24.100539 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "550994e0b3cd4604907782232f565109"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "bf4938a027f65c5d73d3122a45cade89"
server_key_iv: "ee1c90046890d321652514d39c9f1c74"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.101303 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "550994e0b3cd4604907782232f565109"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "bf4938a027f65c5d73d3122a45cade89"
server_key_iv: "ee1c90046890d321652514d39c9f1c74"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.106280 20543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:38:24.109671 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.111075 20543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:38:24.111223 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "550994e0b3cd4604907782232f565109"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "bf4938a027f65c5d73d3122a45cade89"
server_key_iv: "ee1c90046890d321652514d39c9f1c74"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.111418 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:24.117237 20543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:24.118063 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:24.118319 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:24.119232 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:24.120684 20543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:24.120777 20543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.120891 20543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:24.120958 20543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.150334 20543 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:43965
I20260926 16:38:24.150384 20669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:43965 every 8 connection(s)
I20260926 16:38:24.151562 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:38:24.154014 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20543
I20260926 16:38:24.154220 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:38:24.154844 10680 external_mini_cluster.cc:1428] Setting key 9563128a0ddc767759f938006fe0f4a3
I20260926 16:38:24.157481 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32883
--builtin_ntp_servers=127.10.110.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:38:24.159447 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.153920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:36669 (local address 127.10.110.62:32883)
0926 16:38:24.154365 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:24.154371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:24.155605 (+ 1234us) server_negotiation.cc:407] Connection header received
0926 16:38:24.157392 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:24.157398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:24.157457 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:24.157584 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:24.158811 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:24.158816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:24.158820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:24.158838 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:24.158883 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:24.158897 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:24.159000 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:24.159136 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 16:38:24.159189 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260926 16:38:24.160315 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.154465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32883 (local address 127.10.110.1:36669)
0926 16:38:24.155452 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:24.155501 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:24.157144 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:24.157808 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:24.157816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:24.157907 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:24.158535 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:38:24.158537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:24.158573 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:38:24.158655 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:24.158981 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:24.158986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:24.158991 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:38:24.159181 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 16:38:24.159368 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":291,"threads_started":1}
I20260926 16:38:24.161674 20670 heartbeater.cc:347] Connected to a master server at 127.10.110.62:32883
I20260926 16:38:24.161936 20670 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:24.162483 20670 heartbeater.cc:511] Master 127.10.110.62:32883 requested a full tablet report, sending...
I20260926 16:38:24.164356 20484 ts_manager.cc:194] Registered new tserver with Master: 550994e0b3cd4604907782232f565109 (127.10.110.1:43965)
W20260926 16:38:24.467983 20674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:24.468263 20674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:24.468395 20674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:24.475700 20674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:24.475816 20674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:24.475996 20674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:38:24.485028 20674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:24.486284 20674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:24.488067 20674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:24.500392 20679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:24.500424 20682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:24.500463 20680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:24.501289 20674 server_base.cc:1034] running on GCE node
I20260926 16:38:24.501833 20674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:24.502705 20674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:24.503943 20674 hybrid_clock.cc:648] HybridClock initialized: now 1790440704503866 us; error 46 us; skew 500 ppm
I20260926 16:38:24.504245 20674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:24.506363 20674 webserver.cc:460] Webserver started at http://127.10.110.2:42437/ using document root <none> and password file <none>
I20260926 16:38:24.507018 20674 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:24.507113 20674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:24.507409 20674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:24.510218 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "4a3f1d4be2bb47efb8aedde1795a9ce5"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "de4ea25bcd4700c574ab072d06881101"
server_key_iv: "095cb0fbfd816c88717b816a89366006"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.511022 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "4a3f1d4be2bb47efb8aedde1795a9ce5"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "de4ea25bcd4700c574ab072d06881101"
server_key_iv: "095cb0fbfd816c88717b816a89366006"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.515913 20674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:38:24.519196 20688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.520474 20674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:38:24.520605 20674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "4a3f1d4be2bb47efb8aedde1795a9ce5"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "de4ea25bcd4700c574ab072d06881101"
server_key_iv: "095cb0fbfd816c88717b816a89366006"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.520787 20674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:24.526083 20674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:24.526803 20674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:24.527040 20674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:24.528036 20674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:24.529346 20674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:24.529435 20674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.529537 20674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:24.529606 20674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.558553 20674 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:33361
I20260926 16:38:24.558601 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:33361 every 8 connection(s)
I20260926 16:38:24.559775 20674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:38:24.567328 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.562082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:59225 (local address 127.10.110.62:32883)
0926 16:38:24.562230 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:24.562235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:24.563532 (+ 1297us) server_negotiation.cc:407] Connection header received
0926 16:38:24.565266 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:24.565271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:24.565346 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:24.565430 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:24.566629 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:24.566635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:24.566639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:24.566657 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:24.566686 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:24.566697 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:24.566799 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:24.566980 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:38:24.567045 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:38:24.568297 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.562540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32883 (local address 127.10.110.2:59225)
0926 16:38:24.563371 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:24.563413 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:24.565018 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:24.565660 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:24.565668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:24.565755 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:24.566348 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:38:24.566350 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:24.566386 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:38:24.566469 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:24.566778 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:24.566788 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:24.566795 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:38:24.567081 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 16:38:24.567348 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":239,"threads_started":1}
I20260926 16:38:24.568491 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20674
I20260926 16:38:24.568656 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:38:24.569162 10680 external_mini_cluster.cc:1428] Setting key f4648871e76d2aef5e812d072ca23b2b
I20260926 16:38:24.569610 20801 heartbeater.cc:347] Connected to a master server at 127.10.110.62:32883
I20260926 16:38:24.569892 20801 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:24.570495 20801 heartbeater.cc:511] Master 127.10.110.62:32883 requested a full tablet report, sending...
I20260926 16:38:24.571641 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32883
--builtin_ntp_servers=127.10.110.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:38:24.571812 20484 ts_manager.cc:194] Registered new tserver with Master: 4a3f1d4be2bb47efb8aedde1795a9ce5 (127.10.110.2:33361)
W20260926 16:38:24.878975 20804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:24.879290 20804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:24.879387 20804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:24.886444 20804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:24.886541 20804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:24.886648 20804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:38:24.895586 20804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:32883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:24.896818 20804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:24.898459 20804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:24.910598 20812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:24.910665 20810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:24.910876 20809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:24.911728 20804 server_base.cc:1034] running on GCE node
I20260926 16:38:24.912295 20804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:24.913137 20804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:24.914320 20804 hybrid_clock.cc:648] HybridClock initialized: now 1790440704914263 us; error 55 us; skew 500 ppm
I20260926 16:38:24.914603 20804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:38:24.916677 20804 webserver.cc:460] Webserver started at http://127.10.110.3:36599/ using document root <none> and password file <none>
I20260926 16:38:24.917315 20804 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:24.917397 20804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:24.917629 20804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:24.920354 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "38b5656b9acd4254a57882ae08bbe039"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "9300f1f4a3486f83ed880dc5d7b96f74"
server_key_iv: "2fd6badb42cd6454133c42ec3f1d3c42"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.921066 20804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "38b5656b9acd4254a57882ae08bbe039"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "9300f1f4a3486f83ed880dc5d7b96f74"
server_key_iv: "2fd6badb42cd6454133c42ec3f1d3c42"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.926056 20804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:38:24.929466 20818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.930855 20804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:38:24.931013 20804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "38b5656b9acd4254a57882ae08bbe039"
format_stamp: "Formatted at 2026-09-26 16:38:24 on dist-test-slave-whjp"
server_key: "9300f1f4a3486f83ed880dc5d7b96f74"
server_key_iv: "2fd6badb42cd6454133c42ec3f1d3c42"
server_key_version: "encryptionkey@0"
I20260926 16:38:24.931210 20804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:24.936569 20804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:24.937330 20804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:24.937563 20804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:24.938412 20804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:24.939710 20804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:24.939800 20804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.939987 20804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:24.940057 20804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:24.968354 20804 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:39549
I20260926 16:38:24.968420 20930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:39549 every 8 connection(s)
I20260926 16:38:24.969689 20804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:38:24.970932 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20804
I20260926 16:38:24.971117 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:38:24.971508 10680 external_mini_cluster.cc:1428] Setting key b92adbde896245a9c7a227effd93455e
I20260926 16:38:24.977718 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.972153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:41551 (local address 127.10.110.62:32883)
0926 16:38:24.972333 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:24.972339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:24.973878 (+ 1539us) server_negotiation.cc:407] Connection header received
0926 16:38:24.975652 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:24.975657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:24.975718 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:24.975798 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:24.977002 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:24.977008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:24.977013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:24.977029 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:24.977061 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:24.977073 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:24.977177 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:24.977375 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:38:24.977441 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:38:24.978613 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.972793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:32883 (local address 127.10.110.3:41551)
0926 16:38:24.973738 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:24.973775 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:24.975403 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:24.976068 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:24.976076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:24.976167 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:24.976727 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:38:24.976729 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:24.976766 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:38:24.976849 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:24.977157 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:24.977167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:24.977174 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:38:24.977439 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 16:38:24.977692 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":281,"threads_started":1}
I20260926 16:38:24.979929 20931 heartbeater.cc:347] Connected to a master server at 127.10.110.62:32883
I20260926 16:38:24.980197 20931 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:24.980734 20931 heartbeater.cc:511] Master 127.10.110.62:32883 requested a full tablet report, sending...
I20260926 16:38:24.982028 20484 ts_manager.cc:194] Registered new tserver with Master: 38b5656b9acd4254a57882ae08bbe039 (127.10.110.3:39549)
I20260926 16:38:24.986907 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:38:24.992682 20672 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:38:24.991179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.10.110.62:32883)
0926 16:38:24.991324 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:24.991329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:24.991805 (+ 476us) server_negotiation.cc:407] Connection header received
0926 16:38:24.992004 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:24.992007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:24.992056 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:24.992136 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:24.992495 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36618: BlockingRecv error: recv got EOF from 127.0.0.1:36618 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:38:24.993192 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20543
I20260926 16:38:24.996621 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:24.994066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.10.110.1:43965)
0926 16:38:24.994551 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:24.994558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:24.994582 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:38:24.994636 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:24.994642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:24.994850 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:24.994999 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:24.995920 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:24.995927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:24.995938 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:24.995956 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:24.995999 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:24.996014 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:24.996135 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:24.996264 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 16:38:24.996365 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":237,"threads_started":1}
I20260926 16:38:24.997833 20665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:25.076800 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20543
I20260926 16:38:25.090349 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20674
I20260926 16:38:25.093499 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:25.091197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.10.110.2:33361)
0926 16:38:25.091641 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:25.091648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:25.091674 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:38:25.091731 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:25.091737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:25.092000 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:25.092118 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:25.092890 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:25.092897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:25.092908 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:25.092920 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:25.092962 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:25.092980 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:25.093100 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:25.093148 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 16:38:25.093235 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":202,"threads_started":1}
I20260926 16:38:25.094637 20796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:25.174762 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20674
I20260926 16:38:25.186903 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20804
I20260926 16:38:25.190657 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:25.187807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.10.110.3:39549)
0926 16:38:25.188421 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:25.188435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:25.188463 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:38:25.188522 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:25.188529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:25.188753 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:25.188899 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:25.189951 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:25.189957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:25.189967 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:38:25.189983 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:25.190038 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:38:25.190052 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:25.190200 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:25.190272 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 16:38:25.190362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":289,"threads_started":1}
I20260926 16:38:25.191805 20926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:25.272297 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20804
I20260926 16:38:25.285014 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20453
I20260926 16:38:25.286186 20514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:25.384007 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20453
2026-09-26T16:38:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2231 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:38:25 dist-test-slave-whjp krb5kdc[20949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:38:25 dist-test-slave-whjp krb5kdc[20949](info): set up 2 sockets
Sep 26 16:38:25 dist-test-slave-whjp krb5kdc[20949](info): commencing operation
krb5kdc: starting...
W20260926 16:38:26.393306 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:38:26 dist-test-slave-whjp krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:38:26.601285 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33461
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:33461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:26.912045 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:26.912320 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:26.912436 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:26.919392 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:38:26.919502 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:26.919574 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:26.919639 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:38:26.919692 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:38:26.929476 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:33461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:33461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:26.930766 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:26.932112 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:26.942813 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:26.942773 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:26.942839 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:26.943660 20962 server_base.cc:1034] running on GCE node
I20260926 16:38:26.944433 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:26.945706 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:26.946902 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790440706946853 us; error 35 us; skew 500 ppm
I20260926 16:38:26.947219 20962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:26 dist-test-slave-whjp krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440706, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:26.950073 20962 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:38:26.951498 20962 webserver.cc:460] Webserver started at http://127.10.110.62:33065/ using document root <none> and password file <none>
I20260926 16:38:26.952178 20962 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:26.952288 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:26.952558 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:26.955341 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "2702aa4e97d240ce8a236c322ccc5b4f"
format_stamp: "Formatted at 2026-09-26 16:38:26 on dist-test-slave-whjp"
server_key: "522b95d73f1329573f64893248781f9d"
server_key_iv: "6c0ccbbf645f570119106e569f49f8ec"
server_key_version: "encryptionkey@0"
I20260926 16:38:26.956100 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "2702aa4e97d240ce8a236c322ccc5b4f"
format_stamp: "Formatted at 2026-09-26 16:38:26 on dist-test-slave-whjp"
server_key: "522b95d73f1329573f64893248781f9d"
server_key_iv: "6c0ccbbf645f570119106e569f49f8ec"
server_key_version: "encryptionkey@0"
I20260926 16:38:26.960896 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:38:26.964140 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:26.965399 20962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:38:26.965546 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "2702aa4e97d240ce8a236c322ccc5b4f"
format_stamp: "Formatted at 2026-09-26 16:38:26 on dist-test-slave-whjp"
server_key: "522b95d73f1329573f64893248781f9d"
server_key_iv: "6c0ccbbf645f570119106e569f49f8ec"
server_key_version: "encryptionkey@0"
I20260926 16:38:26.965721 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:26.977242 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:26.981328 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:26.981590 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:26.998348 20962 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:33461
I20260926 16:38:26.998377 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:33461 every 8 connection(s)
I20260926 16:38:26.999670 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:38:27.000717 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 20962
I20260926 16:38:27.000926 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:38:27.001694 10680 external_mini_cluster.cc:1428] Setting key 7801bffd1539037d154ea318625235b7
I20260926 16:38:27.004235 21029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:38:27 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:27.016754 21029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f: Bootstrap starting.
I20260926 16:38:27.020834 21029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:27.022087 21029 log.cc:824] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:27.024562 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:27.003701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.10.110.62:33461)
0926 16:38:27.004477 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:27.004499 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:27.004539 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:38:27.006039 (+ 1500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:27.006060 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:27.006427 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:27.006761 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:27.008416 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:27.009724 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:27.011443 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:27.012136 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:27.012255 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:27.016655 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:27.016667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:27.016688 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:27.016739 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:27.020372 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:27.021049 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:27.021058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:27.021067 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:27.021291 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:27.021627 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:27.021631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:27.021634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:27.022309 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:27.022615 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:27.022923 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 16:38:27.023273 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260926 16:38:27.025506 21029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f: No bootstrap required, opened a new log
I20260926 16:38:27.032853 21029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33461 } }
I20260926 16:38:27.033197 21029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:27.033268 21029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2702aa4e97d240ce8a236c322ccc5b4f, State: Initialized, Role: FOLLOWER
I20260926 16:38:27.033776 21029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33461 } }
I20260926 16:38:27.033981 21029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:38:27.034076 21029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:38:27.034194 21029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:27.036404 21029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33461 } }
I20260926 16:38:27.036834 21029 leader_election.cc:302] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2702aa4e97d240ce8a236c322ccc5b4f; no voters:
I20260926 16:38:27.037343 21029 leader_election.cc:288] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:38:27.037460 21034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:38:27.038053 21034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [term 1 LEADER]: Becoming Leader. State: Replica: 2702aa4e97d240ce8a236c322ccc5b4f, State: Running, Role: LEADER
I20260926 16:38:27.038838 21034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33461 } }
I20260926 16:38:27.039218 21029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:38:27.041348 21036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2702aa4e97d240ce8a236c322ccc5b4f. Latest consensus state: current_term: 1 leader_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33461 } } }
I20260926 16:38:27.041689 21036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:27.041585 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2702aa4e97d240ce8a236c322ccc5b4f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 33461 } } }
I20260926 16:38:27.041903 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f [sys.catalog]: This master's current role is: LEADER
I20260926 16:38:27.043587 21041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:38:27.048111 21041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:38:27.058346 21041 catalog_manager.cc:1347] Generated new cluster ID: f34187717661442e975d05c6d89a25e6
I20260926 16:38:27.058490 21041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:38:27.089989 21041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:38:27.091886 21041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:38:27.104980 21041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f: Generated new TSK 0
I20260926 16:38:27.106030 21041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:38:27.164552 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33461
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:27.477952 21055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:27.478215 21055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:27.478304 21055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:27.485391 21055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:27.485497 21055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:27.485613 21055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:38:27.494989 21055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:27.496170 21055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:27.497347 21055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:27.509227 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:27.509207 21061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:27.509696 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:27.509989 21055 server_base.cc:1034] running on GCE node
I20260926 16:38:27.510565 21055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:27.511471 21055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:27.512657 21055 hybrid_clock.cc:648] HybridClock initialized: now 1790440707512613 us; error 35 us; skew 500 ppm
I20260926 16:38:27.512966 21055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:27 dist-test-slave-whjp krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440707, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:27.519474 21055 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:38:27.520828 21055 webserver.cc:460] Webserver started at http://127.10.110.1:39279/ using document root <none> and password file <none>
I20260926 16:38:27.521476 21055 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:27.521585 21055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:27.521843 21055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:27.524602 21055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "cfb2a10168344102bae18007ac99f906"
format_stamp: "Formatted at 2026-09-26 16:38:27 on dist-test-slave-whjp"
server_key: "359a0979a774f95a83c837c7df234bc4"
server_key_iv: "8f8ba56e46b7261d24f9a9ceb47c7472"
server_key_version: "encryptionkey@0"
I20260926 16:38:27.525296 21055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "cfb2a10168344102bae18007ac99f906"
format_stamp: "Formatted at 2026-09-26 16:38:27 on dist-test-slave-whjp"
server_key: "359a0979a774f95a83c837c7df234bc4"
server_key_iv: "8f8ba56e46b7261d24f9a9ceb47c7472"
server_key_version: "encryptionkey@0"
I20260926 16:38:27.529927 21055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 16:38:27.533203 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:27.534524 21055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 16:38:27.534662 21055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "cfb2a10168344102bae18007ac99f906"
format_stamp: "Formatted at 2026-09-26 16:38:27 on dist-test-slave-whjp"
server_key: "359a0979a774f95a83c837c7df234bc4"
server_key_iv: "8f8ba56e46b7261d24f9a9ceb47c7472"
server_key_version: "encryptionkey@0"
I20260926 16:38:27.534821 21055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:27.547128 21055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:27.550848 21055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:27.551115 21055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:27.551988 21055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:27.553296 21055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:27.553373 21055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:27.553485 21055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:27.553537 21055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:27.582273 21055 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:40179
I20260926 16:38:27.582312 21182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:40179 every 8 connection(s)
I20260926 16:38:27.583456 21055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:38:27.584329 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 21055
I20260926 16:38:27.584532 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:38:27.585290 10680 external_mini_cluster.cc:1428] Setting key 1fb023538d5ed370a9e21dedf50961ee
Sep 26 16:38:27 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440707, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:27.603662 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:27.586133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:42547 (local address 127.10.110.62:33461)
0926 16:38:27.586549 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:27.586553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:27.587683 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 16:38:27.590049 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:27.590053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:27.590115 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:27.590203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:27.591990 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:27.592667 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:27.594652 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:27.595046 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:27.595150 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:27.598476 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:27.598481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:27.598485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:27.598519 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:27.600784 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:27.601532 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:27.601535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:27.601537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:27.601639 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:27.602025 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:27.602028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:27.602030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:27.602279 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:27.602380 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:27.603178 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 16:38:27.603342 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":220,"threads_started":1}
I20260926 16:38:27.604861 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:27.586677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33461 (local address 127.10.110.1:42547)
0926 16:38:27.587521 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:27.587573 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:27.589748 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:27.590436 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:27.590446 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:27.591075 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:27.591794 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:27.591816 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:27.592807 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:27.592812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:27.594538 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:27.594552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:27.595239 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:27.595245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:27.595430 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:27.596090 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:27.596111 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:27.598259 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:27.600905 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:27.600914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:27.600929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:27.601435 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:27.601719 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:27.601724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:27.601727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:27.601935 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:27.602384 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:27.602392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:27.602774 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 16:38:27.603061 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 16:38:27.603581 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":345,"threads_started":1}
I20260926 16:38:27.606316 21183 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33461
I20260926 16:38:27.606668 21183 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:27.607599 21183 heartbeater.cc:511] Master 127.10.110.62:33461 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:38:27.609792 20994 ts_manager.cc:194] Registered new tserver with Master: cfb2a10168344102bae18007ac99f906 (127.10.110.1:40179)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:38:27.611240 20994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:42547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:38:27.634785 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33461
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:27.948112 21189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:27.948369 21189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:27.948459 21189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:27.955538 21189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:27.955633 21189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:27.955740 21189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:38:27.965080 21189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:27.966267 21189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:27.967530 21189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:27.979607 21195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:27.979696 21194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:27.979607 21197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:27.980774 21189 server_base.cc:1034] running on GCE node
I20260926 16:38:27.981372 21189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:27.982270 21189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:27.983474 21189 hybrid_clock.cc:648] HybridClock initialized: now 1790440707983427 us; error 40 us; skew 500 ppm
I20260926 16:38:27.983794 21189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:27 dist-test-slave-whjp krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440707, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:27.986712 21189 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:38:27.988121 21189 webserver.cc:460] Webserver started at http://127.10.110.2:37877/ using document root <none> and password file <none>
I20260926 16:38:27.988765 21189 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:27.988857 21189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:27.989104 21189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:27.991981 21189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "7247ecdbf8bc433f8d2bf64558753507"
format_stamp: "Formatted at 2026-09-26 16:38:27 on dist-test-slave-whjp"
server_key: "58f6bde95b0820fefc309ad1077dd182"
server_key_iv: "436fe74aeecdf435770fb1bd1976029d"
server_key_version: "encryptionkey@0"
I20260926 16:38:27.992694 21189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "7247ecdbf8bc433f8d2bf64558753507"
format_stamp: "Formatted at 2026-09-26 16:38:27 on dist-test-slave-whjp"
server_key: "58f6bde95b0820fefc309ad1077dd182"
server_key_iv: "436fe74aeecdf435770fb1bd1976029d"
server_key_version: "encryptionkey@0"
I20260926 16:38:27.997812 21189 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:38:28.001268 21204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:28.002625 21189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:38:28.002766 21189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "7247ecdbf8bc433f8d2bf64558753507"
format_stamp: "Formatted at 2026-09-26 16:38:27 on dist-test-slave-whjp"
server_key: "58f6bde95b0820fefc309ad1077dd182"
server_key_iv: "436fe74aeecdf435770fb1bd1976029d"
server_key_version: "encryptionkey@0"
I20260926 16:38:28.002933 21189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:28.015523 21189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:28.019237 21189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:28.019500 21189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:28.020465 21189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:28.021720 21189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:28.021793 21189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:28.021908 21189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:28.021961 21189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:28.050942 21189 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:40673
I20260926 16:38:28.050989 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:40673 every 8 connection(s)
I20260926 16:38:28.052191 21189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:38:28.054921 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 21189
I20260926 16:38:28.055099 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:38:28.055558 10680 external_mini_cluster.cc:1428] Setting key 72dc97c371220ad4d61ab0fb2d57fba8
Sep 26 16:38:28 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440707, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:28.073038 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.054563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:33213 (local address 127.10.110.62:33461)
0926 16:38:28.054764 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.054770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.056036 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 16:38:28.058117 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.058121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.058182 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.058256 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:28.060101 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.060787 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.062754 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.063181 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.063269 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.067277 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:28.067284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:28.067289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:28.067326 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:28.070259 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.070930 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.070935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.070938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.071089 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.071436 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.071440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.071443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.071734 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:28.071881 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.072510 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.072711 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:38:28.073915 21319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.055069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33461 (local address 127.10.110.2:33213)
0926 16:38:28.055874 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.055926 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.057846 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.058511 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.058522 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.059175 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:28.059955 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.059979 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.060930 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.060936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.062617 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.062631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.063383 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.063390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.063728 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.064471 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:28.064498 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:28.067050 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:28.070382 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:28.070388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:28.070401 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:28.070815 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:28.071174 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:28.071177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:28.071179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:28.071346 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:28.071862 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:28.071870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:28.072180 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.072389 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.072765 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":278,"threads_started":1}
I20260926 16:38:28.075212 21317 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33461
I20260926 16:38:28.075481 21317 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:28.076201 21317 heartbeater.cc:511] Master 127.10.110.62:33461 requested a full tablet report, sending...
I20260926 16:38:28.077601 20994 ts_manager.cc:194] Registered new tserver with Master: 7247ecdbf8bc433f8d2bf64558753507 (127.10.110.2:40673)
I20260926 16:38:28.078377 20994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:33213
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:38:28.104955 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33461
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:38:28.425881 21322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:38:28.426162 21322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:38:28.426293 21322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:38:28.433534 21322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:38:28.433645 21322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:38:28.433794 21322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:38:28.443964 21322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:33461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:38:28.445348 21322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:38:28.446667 21322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:38:28.459462 21327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:28.459533 21328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:38:28.459494 21330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:38:28.460374 21322 server_base.cc:1034] running on GCE node
I20260926 16:38:28.461009 21322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:38:28.461977 21322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:38:28.463192 21322 hybrid_clock.cc:648] HybridClock initialized: now 1790440708463137 us; error 48 us; skew 500 ppm
I20260926 16:38:28.463514 21322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:38:28 dist-test-slave-whjp krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440708, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:38:28.466586 21322 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:38:28.468046 21322 webserver.cc:460] Webserver started at http://127.10.110.3:36037/ using document root <none> and password file <none>
I20260926 16:38:28.468742 21322 fs_manager.cc:359] Metadata directory not provided
I20260926 16:38:28.468852 21322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:38:28.469132 21322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:38:28.472101 21322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "8c630ea0fc9840d581ebf25cb938bde7"
format_stamp: "Formatted at 2026-09-26 16:38:28 on dist-test-slave-whjp"
server_key: "158d648b4cd08574a490aac4bd8779e9"
server_key_iv: "bd80bc4367d38558223b5ba01df78268"
server_key_version: "encryptionkey@0"
I20260926 16:38:28.472886 21322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "8c630ea0fc9840d581ebf25cb938bde7"
format_stamp: "Formatted at 2026-09-26 16:38:28 on dist-test-slave-whjp"
server_key: "158d648b4cd08574a490aac4bd8779e9"
server_key_iv: "bd80bc4367d38558223b5ba01df78268"
server_key_version: "encryptionkey@0"
I20260926 16:38:28.477978 21322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:38:28.481411 21337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:28.482688 21322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:38:28.482836 21322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "8c630ea0fc9840d581ebf25cb938bde7"
format_stamp: "Formatted at 2026-09-26 16:38:28 on dist-test-slave-whjp"
server_key: "158d648b4cd08574a490aac4bd8779e9"
server_key_iv: "bd80bc4367d38558223b5ba01df78268"
server_key_version: "encryptionkey@0"
I20260926 16:38:28.483016 21322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:38:28.506673 21322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:38:28.510512 21322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:38:28.510788 21322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:38:28.511715 21322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:38:28.513010 21322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:38:28.513087 21322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:28.513199 21322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:38:28.513255 21322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:38:28.542696 21322 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:46307
I20260926 16:38:28.542757 21449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:46307 every 8 connection(s)
I20260926 16:38:28.544006 21322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:38:28.545375 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 21322
I20260926 16:38:28.545584 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:38:28.546108 10680 external_mini_cluster.cc:1428] Setting key 3fa74ea166faaf5e8eba80ee97ad53c3
Sep 26 16:38:28 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440708, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:38:28.563735 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.546625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:59777 (local address 127.10.110.62:33461)
0926 16:38:28.546786 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.546790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.548243 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 16:38:28.550048 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.550052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.550117 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.550207 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:28.551640 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.552413 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.554066 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.554457 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.554556 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.558009 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:28.558015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:28.558019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:28.558054 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:28.560655 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.561419 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.561424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.561426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.561543 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.561988 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.561991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.561994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.562283 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:28.562416 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.563150 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.563376 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:38:28.564883 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.547248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:33461 (local address 127.10.110.3:59777)
0926 16:38:28.548115 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.548159 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.549785 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.550386 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.550393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.550916 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:38:28.551485 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.551503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.552566 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.552570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.553933 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.553955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.554630 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.554634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.554843 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.555499 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:38:28.555523 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:38:28.557767 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:38:28.560810 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:28.560820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:28.560841 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:28.561300 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:28.561650 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:38:28.561655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:38:28.561659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:38:28.561859 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:38:28.562421 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:38:28.562430 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:38:28.562800 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.563056 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.563645 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":298,"threads_started":1}
I20260926 16:38:28.566190 21450 heartbeater.cc:347] Connected to a master server at 127.10.110.62:33461
I20260926 16:38:28.566474 21450 heartbeater.cc:464] Registering TS with master...
I20260926 16:38:28.567196 21450 heartbeater.cc:511] Master 127.10.110.62:33461 requested a full tablet report, sending...
I20260926 16:38:28.568661 20994 ts_manager.cc:194] Registered new tserver with Master: 8c630ea0fc9840d581ebf25cb938bde7 (127.10.110.3:46307)
I20260926 16:38:28.569574 20994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:59777
I20260926 16:38:28.573556 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:38:28.591312 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.578115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.10.110.62:33461)
0926 16:38:28.578296 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.578302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.578700 (+ 398us) server_negotiation.cc:407] Connection header received
0926 16:38:28.578865 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.578869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.578936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.579028 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:28.580427 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.581152 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.582482 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.583140 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.583268 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.585885 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:28.585892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:28.585898 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:28.585942 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:28.588700 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.589465 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.589469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.589472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.589608 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.590081 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.590085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.590087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.590416 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:28.590593 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.590882 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.591045 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:38:28.595438 20994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:38:28.602130 20994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:38:28.613998 21183 heartbeater.cc:503] Master 127.10.110.62:33461 was elected leader, sending a full tablet report...
I20260926 16:38:28.639274 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.614324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:40673 (local address 127.0.0.1:56210)
0926 16:38:28.625053 (+ 10729us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.625117 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.625355 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.630455 (+ 5100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.630462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.630498 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:38:28.631067 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.631090 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.633608 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.633615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.636263 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.636281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.638181 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.638188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.638402 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.638480 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.638595 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.638794 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10406,"mutex_wait_us":3,"thread_start_us":10269,"threads_started":1}
I20260926 16:38:28.639552 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.615196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.10.110.3:46307)
0926 16:38:28.624530 (+ 9334us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.624544 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.630325 (+ 5781us) server_negotiation.cc:407] Connection header received
0926 16:38:28.630467 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.630478 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.630732 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.631081 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:38:28.632319 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.633902 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.636457 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.637955 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.638083 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.638326 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.639013 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.639192 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9103,"thread_start_us":8912,"threads_started":1}
I20260926 16:38:28.640483 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.614414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:46307 (local address 127.0.0.1:39542)
0926 16:38:28.629773 (+ 15359us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.629990 (+ 217us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.630161 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.631399 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.631404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.631431 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:38:28.632073 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.632089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.634061 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.634065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.636276 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.636285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.638495 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.638501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.638664 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.638693 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.638754 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.638856 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15214,"mutex_wait_us":57,"thread_start_us":14894,"threads_started":1}
I20260926 16:38:28.640851 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.614573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.10.110.2:40673)
0926 16:38:28.625602 (+ 11029us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.625612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.629215 (+ 3603us) server_negotiation.cc:407] Connection header received
0926 16:38:28.629434 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.629445 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.630139 (+ 694us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.631368 (+ 1229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:38:28.631600 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.633451 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.636462 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.637975 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.640115 (+ 2140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.640328 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.640402 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.640590 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10860,"thread_start_us":10759,"threads_started":1}
I20260926 16:38:28.645488 21385 tablet_service.cc:1467] Processing CreateTablet for tablet dd9b6b7ad07440e5b827d83fc07470dd (DEFAULT_TABLE table=test-table [id=012e690964ac40b392fcf4643743c4e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:38:28.645493 21252 tablet_service.cc:1467] Processing CreateTablet for tablet dd9b6b7ad07440e5b827d83fc07470dd (DEFAULT_TABLE table=test-table [id=012e690964ac40b392fcf4643743c4e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:38:28.646966 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9b6b7ad07440e5b827d83fc07470dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:38:28.646977 21385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9b6b7ad07440e5b827d83fc07470dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:38:28.648176 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.630368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:40179 (local address 127.0.0.1:59386)
0926 16:38:28.639536 (+ 9168us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.639557 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.639663 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.642912 (+ 3249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.642917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.642954 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:38:28.643126 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.643136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.644555 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.644560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.646127 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.646141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.647419 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.647425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.647600 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.647632 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.647718 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.647894 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9063,"thread_start_us":10963,"threads_started":1}
I20260926 16:38:28.648231 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.630396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.10.110.1:40179)
0926 16:38:28.641960 (+ 11564us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.641970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.642001 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:38:28.642107 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.642116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.642378 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.642923 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:38:28.643331 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.644420 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.646297 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.647270 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.647388 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.647618 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.647769 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.647954 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11386,"thread_start_us":11256,"threads_started":1}
I20260926 16:38:28.653357 21118 tablet_service.cc:1467] Processing CreateTablet for tablet dd9b6b7ad07440e5b827d83fc07470dd (DEFAULT_TABLE table=test-table [id=012e690964ac40b392fcf4643743c4e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:38:28.654731 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd9b6b7ad07440e5b827d83fc07470dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:38:28.661844 21468 tablet_bootstrap.cc:492] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Bootstrap starting.
I20260926 16:38:28.662689 21469 tablet_bootstrap.cc:492] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Bootstrap starting.
I20260926 16:38:28.666146 21468 tablet_bootstrap.cc:654] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:28.666574 21469 tablet_bootstrap.cc:654] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:28.667349 21470 tablet_bootstrap.cc:492] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Bootstrap starting.
I20260926 16:38:28.667340 21468 log.cc:824] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:28.667760 21469 log.cc:824] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:28.670499 21468 tablet_bootstrap.cc:492] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: No bootstrap required, opened a new log
I20260926 16:38:28.670760 21468 ts_tablet_manager.cc:1397] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:38:28.670902 21469 tablet_bootstrap.cc:492] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: No bootstrap required, opened a new log
I20260926 16:38:28.670959 21470 tablet_bootstrap.cc:654] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Neither blocks nor log segments found. Creating new log.
I20260926 16:38:28.671175 21469 ts_tablet_manager.cc:1397] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:38:28.672838 21470 log.cc:824] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Log is configured to *not* fsync() on all Append() calls
I20260926 16:38:28.675908 21470 tablet_bootstrap.cc:492] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: No bootstrap required, opened a new log
I20260926 16:38:28.676187 21470 ts_tablet_manager.cc:1397] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 16:38:28.678632 21468 raft_consensus.cc:348] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.679773 21468 raft_consensus.cc:374] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:28.679915 21468 raft_consensus.cc:729] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c630ea0fc9840d581ebf25cb938bde7, State: Initialized, Role: FOLLOWER
I20260926 16:38:28.680540 21468 consensus_queue.cc:260] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.680792 21469 raft_consensus.cc:348] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.681283 21469 raft_consensus.cc:374] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:28.681582 21450 heartbeater.cc:503] Master 127.10.110.62:33461 was elected leader, sending a full tablet report...
I20260926 16:38:28.682112 21468 ts_tablet_manager.cc:1428] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260926 16:38:28.682268 21469 raft_consensus.cc:729] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7247ecdbf8bc433f8d2bf64558753507, State: Initialized, Role: FOLLOWER
I20260926 16:38:28.682886 21469 consensus_queue.cc:260] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.686160 21470 raft_consensus.cc:348] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.686597 21470 raft_consensus.cc:374] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:38:28.686679 21469 ts_tablet_manager.cc:1428] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 16:38:28.686810 21470 raft_consensus.cc:729] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb2a10168344102bae18007ac99f906, State: Initialized, Role: FOLLOWER
I20260926 16:38:28.687081 21317 heartbeater.cc:503] Master 127.10.110.62:33461 was elected leader, sending a full tablet report...
I20260926 16:38:28.687505 21470 consensus_queue.cc:260] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.689378 21470 ts_tablet_manager.cc:1428] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260926 16:38:28.795475 21451 tablet.cc:2369] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:38:28.800634 21475 raft_consensus.cc:482] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:38:28.800910 21475 raft_consensus.cc:504] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.802495 21475 leader_election.cc:288] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c630ea0fc9840d581ebf25cb938bde7 (127.10.110.3:46307), cfb2a10168344102bae18007ac99f906 (127.10.110.1:40179)
W20260926 16:38:28.804036 21318 tablet.cc:2369] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:38:28.809391 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.802939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:47263 (local address 127.10.110.3:46307)
0926 16:38:28.803116 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.803121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.803666 (+ 545us) server_negotiation.cc:407] Connection header received
0926 16:38:28.803958 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.803965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.804049 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.804116 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:38:28.804952 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.805678 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.807693 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.808518 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.808668 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.808743 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.809029 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.809151 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:38:28.809468 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.802854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:46307 (local address 127.10.110.2:47263)
0926 16:38:28.803565 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.803583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.803722 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.804330 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.804335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.804360 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:38:28.804752 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.804763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.805807 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.805812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.807536 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.807548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.808719 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.808725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.808873 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.808897 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.808976 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.809105 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":356,"threads_started":1}
I20260926 16:38:28.810050 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.803638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:40179 (local address 127.10.110.2:48461)
0926 16:38:28.804562 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 16:38:28.804579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:38:28.804717 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:38:28.805026 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:38:28.805030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:38:28.805051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:38:28.805322 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.805330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.806565 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.806569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.808068 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:38:28.808080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.809335 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.809342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:38:28.809535 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.809570 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:38:28.809694 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:38:28.809811 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":586,"threads_started":1}
I20260926 16:38:28.810129 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.803885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:48461 (local address 127.10.110.1:40179)
0926 16:38:28.804080 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.804086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.804681 (+ 595us) server_negotiation.cc:407] Connection header received
0926 16:38:28.804849 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.804855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.804930 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.805013 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:38:28.805451 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.806403 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.808214 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.809113 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.809282 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.809351 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.809756 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.809891 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:38:28.810582 21405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b6b7ad07440e5b827d83fc07470dd" candidate_uuid: "7247ecdbf8bc433f8d2bf64558753507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c630ea0fc9840d581ebf25cb938bde7" is_pre_election: true
I20260926 16:38:28.811136 21405 raft_consensus.cc:2454] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7247ecdbf8bc433f8d2bf64558753507 in term 0.
I20260926 16:38:28.811110 21138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b6b7ad07440e5b827d83fc07470dd" candidate_uuid: "7247ecdbf8bc433f8d2bf64558753507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb2a10168344102bae18007ac99f906" is_pre_election: true
I20260926 16:38:28.811631 21138 raft_consensus.cc:2454] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7247ecdbf8bc433f8d2bf64558753507 in term 0.
I20260926 16:38:28.811870 21206 leader_election.cc:302] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7247ecdbf8bc433f8d2bf64558753507, 8c630ea0fc9840d581ebf25cb938bde7; no voters:
I20260926 16:38:28.812263 21475 raft_consensus.cc:2781] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:38:28.812387 21475 raft_consensus.cc:482] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:38:28.812489 21475 raft_consensus.cc:3037] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:28.814543 21475 raft_consensus.cc:504] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.815074 21475 leader_election.cc:288] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [CANDIDATE]: Term 1 election: Requested vote from peers 8c630ea0fc9840d581ebf25cb938bde7 (127.10.110.3:46307), cfb2a10168344102bae18007ac99f906 (127.10.110.1:40179)
I20260926 16:38:28.815485 21405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b6b7ad07440e5b827d83fc07470dd" candidate_uuid: "7247ecdbf8bc433f8d2bf64558753507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c630ea0fc9840d581ebf25cb938bde7"
I20260926 16:38:28.815536 21138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd9b6b7ad07440e5b827d83fc07470dd" candidate_uuid: "7247ecdbf8bc433f8d2bf64558753507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb2a10168344102bae18007ac99f906"
I20260926 16:38:28.815655 21405 raft_consensus.cc:3037] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:28.815699 21138 raft_consensus.cc:3037] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:38:28.817724 21138 raft_consensus.cc:2454] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7247ecdbf8bc433f8d2bf64558753507 in term 1.
I20260926 16:38:28.817724 21405 raft_consensus.cc:2454] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7247ecdbf8bc433f8d2bf64558753507 in term 1.
I20260926 16:38:28.818063 21206 leader_election.cc:302] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7247ecdbf8bc433f8d2bf64558753507, cfb2a10168344102bae18007ac99f906; no voters:
I20260926 16:38:28.818288 21475 raft_consensus.cc:2781] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:38:28.818714 21475 raft_consensus.cc:686] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 1 LEADER]: Becoming Leader. State: Replica: 7247ecdbf8bc433f8d2bf64558753507, State: Running, Role: LEADER
I20260926 16:38:28.819226 21475 consensus_queue.cc:237] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } }
I20260926 16:38:28.823980 20994 catalog_manager.cc:5495] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7247ecdbf8bc433f8d2bf64558753507 (127.10.110.2). New cstate: current_term: 1 leader_uuid: "7247ecdbf8bc433f8d2bf64558753507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7247ecdbf8bc433f8d2bf64558753507" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 40673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 } health_report { overall_health: UNKNOWN } } }
W20260926 16:38:28.835891 21184 tablet.cc:2369] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:38:28.881701 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.876557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.10.110.2:40673)
0926 16:38:28.876757 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.876763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.876900 (+ 137us) server_negotiation.cc:407] Connection header received
0926 16:38:28.877123 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.877128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.877219 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.877360 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:38:28.877745 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.878431 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.879729 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.880146 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.880235 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.880694 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:38:28.881190 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:38:28.881294 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.881396 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.881462 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:38:28.889940 21405 raft_consensus.cc:1261] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 1 FOLLOWER]: Refusing update from remote peer 7247ecdbf8bc433f8d2bf64558753507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:38:28.889940 21138 raft_consensus.cc:1261] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 1 FOLLOWER]: Refusing update from remote peer 7247ecdbf8bc433f8d2bf64558753507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:38:28.890854 21479 consensus_queue.cc:1035] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb2a10168344102bae18007ac99f906" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:38:28.891216 21475 consensus_queue.cc:1035] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c630ea0fc9840d581ebf25cb938bde7" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:38:28.902298 21482 mvcc.cc:205] Tried to move back new op lower bound from 7333645143603503104 to 7333645143328231424. Current Snapshot: MvccSnapshot[applied={T|T < 7333645143603503104}]
I20260926 16:38:28.902897 21484 mvcc.cc:205] Tried to move back new op lower bound from 7333645143603503104 to 7333645143328231424. Current Snapshot: MvccSnapshot[applied={T|T < 7333645143603503104}]
I20260926 16:38:28.904554 21483 mvcc.cc:205] Tried to move back new op lower bound from 7333645143603503104 to 7333645143328231424. Current Snapshot: MvccSnapshot[applied={T|T < 7333645143603503104}]
I20260926 16:38:28.916616 20993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:38:28.916903 20993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:38:28.920925 20993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2702aa4e97d240ce8a236c322ccc5b4f: Sending DeleteTablet for 3 replicas of tablet dd9b6b7ad07440e5b827d83fc07470dd
I20260926 16:38:28.922212 21252 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b6b7ad07440e5b827d83fc07470dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:38:28 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:56210
I20260926 16:38:28.922559 21118 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b6b7ad07440e5b827d83fc07470dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:38:28 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:59386
I20260926 16:38:28.923035 21385 tablet_service.cc:1514] Processing DeleteTablet for tablet dd9b6b7ad07440e5b827d83fc07470dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:38:28 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:39542
I20260926 16:38:28.923631 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 21055
I20260926 16:38:28.938433 21494 tablet_replica.cc:330] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: stopping tablet replica
I20260926 16:38:28.939168 21492 tablet_replica.cc:330] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: stopping tablet replica
I20260926 16:38:28.939476 21493 tablet_replica.cc:330] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: stopping tablet replica
I20260926 16:38:28.939708 21494 raft_consensus.cc:2229] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:38:28.939930 21492 raft_consensus.cc:2229] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:38:28.939954 21493 raft_consensus.cc:2229] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:38:28.940294 21494 raft_consensus.cc:2258] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:38:28.940595 21493 raft_consensus.cc:2258] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:38:28.942968 21493 ts_tablet_manager.cc:1905] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:38:28.943022 21494 ts_tablet_manager.cc:1905] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:38:28.943106 21492 raft_consensus.cc:2258] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:38:28.946722 21492 ts_tablet_manager.cc:1905] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:38:28.950258 21494 ts_tablet_manager.cc:1918] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:38:28.950491 21494 log.cc:1196] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/dd9b6b7ad07440e5b827d83fc07470dd
I20260926 16:38:28.950503 21493 ts_tablet_manager.cc:1918] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:38:28.950732 21493 log.cc:1196] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/dd9b6b7ad07440e5b827d83fc07470dd
I20260926 16:38:28.951050 21494 ts_tablet_manager.cc:1939] T dd9b6b7ad07440e5b827d83fc07470dd P 8c630ea0fc9840d581ebf25cb938bde7: Deleting consensus metadata
I20260926 16:38:28.951306 21493 ts_tablet_manager.cc:1939] T dd9b6b7ad07440e5b827d83fc07470dd P cfb2a10168344102bae18007ac99f906: Deleting consensus metadata
I20260926 16:38:28.953039 20979 catalog_manager.cc:4841] TS 8c630ea0fc9840d581ebf25cb938bde7 (127.10.110.3:46307): tablet dd9b6b7ad07440e5b827d83fc07470dd (table test-table [id=012e690964ac40b392fcf4643743c4e5]) successfully deleted
I20260926 16:38:28.953388 20979 catalog_manager.cc:4841] TS cfb2a10168344102bae18007ac99f906 (127.10.110.1:40179): tablet dd9b6b7ad07440e5b827d83fc07470dd (table test-table [id=012e690964ac40b392fcf4643743c4e5]) successfully deleted
I20260926 16:38:28.957082 21492 ts_tablet_manager.cc:1918] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:38:28.957257 21492 log.cc:1196] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/dd9b6b7ad07440e5b827d83fc07470dd
I20260926 16:38:28.957695 21492 ts_tablet_manager.cc:1939] T dd9b6b7ad07440e5b827d83fc07470dd P 7247ecdbf8bc433f8d2bf64558753507: Deleting consensus metadata
Sep 26 16:38:28 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:38:28.959263 20978 catalog_manager.cc:4841] TS 7247ecdbf8bc433f8d2bf64558753507 (127.10.110.2:40673): tablet dd9b6b7ad07440e5b827d83fc07470dd (table test-table [id=012e690964ac40b392fcf4643743c4e5]) successfully deleted
I20260926 16:38:28.964706 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:28.924578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.10.110.1:40179)
0926 16:38:28.924767 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:28.924774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:28.935262 (+ 10488us) server_negotiation.cc:407] Connection header received
0926 16:38:28.935798 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:28.935806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:28.935955 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:28.936042 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:28.940714 (+ 4672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.941885 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.947025 (+ 5140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:28.947702 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:28.947874 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:28.959047 (+ 11173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:28.959067 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:28.959087 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:28.959134 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:28.961856 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.962753 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.962760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.962762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.962896 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:28.963278 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:28.963281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:28.963283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:28.963637 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:28.963900 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:28.964256 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 16:38:28.964419 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:38:28.965287 21178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:29.068352 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 21055
I20260926 16:38:29.083408 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 21189
Sep 26 16:38:29 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:38:29.100764 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:29.084146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.10.110.2:40673)
0926 16:38:29.084300 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:29.084305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:29.084364 (+ 59us) server_negotiation.cc:407] Connection header received
0926 16:38:29.084589 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:29.084592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:29.084657 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:29.084738 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:29.086359 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:29.087249 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:29.089052 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:29.089491 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:29.089590 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:29.094456 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:29.094470 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:29.094486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:29.094541 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:29.097694 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:29.098433 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:29.098439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:29.098441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:29.098613 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:29.098977 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:29.098980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:29.098982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:29.099298 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:29.099557 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:29.099992 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 16:38:29.100337 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:38:29.101434 21312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:29.212631 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 21189
I20260926 16:38:29.227541 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 21322
Sep 26 16:38:29 dist-test-slave-whjp krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:38:29.241274 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:38:29.228203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.10.110.3:46307)
0926 16:38:29.228402 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:38:29.228407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:38:29.228424 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:38:29.228596 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:38:29.228600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:38:29.228672 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:38:29.228752 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:38:29.229898 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:29.230605 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:29.231736 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:38:29.232325 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:38:29.232420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:38:29.236364 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:38:29.236379 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:38:29.236394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:38:29.236427 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:38:29.238863 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:29.239527 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:29.239534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:29.239537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:29.239676 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:38:29.239997 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:38:29.240001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:38:29.240004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:38:29.240364 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:38:29.240591 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 16:38:29.240809 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 16:38:29.240961 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:38:29.241873 21445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:29.329155 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 21322
I20260926 16:38:29.342320 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 20962
I20260926 16:38:29.343429 21024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:38:29.454525 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 20962
2026-09-26T16:38:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4073 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:38:29 dist-test-slave-whjp krb5kdc[21502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:38:29 dist-test-slave-whjp krb5kdc[21502](info): set up 2 sockets
Sep 26 16:38:29 dist-test-slave-whjp krb5kdc[21502](info): commencing operation
krb5kdc: starting...
W20260926 16:38:30.469286 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:38:30 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:38:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/rangeradmin_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/rangeradmin_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/rangerlookup_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/rangerlookup_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
I20260926 16:38:30.780475 10680 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiK5LtG/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/postgres -l logfile start
2026-09-26 16:38:33.113 UTC [21532] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:38:33.113 UTC [21532] LOG: listening on IPv4 address "127.10.110.20", port 49095
2026-09-26 16:38:33.117 UTC [21532] LOG: listening on Unix socket "/tmp/.s.PGSQL.49095"
2026-09-26 16:38:33.138 UTC [21535] LOG: database system was shut down at 2026-09-26 16:38:31 UTC
2026-09-26 16:38:33.141 UTC [21532] LOG: database system is ready to accept connections
I20260926 16:38:34.044925 10680 mini_postgres.cc:96] Postgres bound to 49095
2026-09-26 16:38:34.060 UTC [21543] FATAL: database "slave" does not exist
127.10.110.20:49095 - accepting connections
I20260926 16:38:34.061628 10680 mini_ranger.cc:161] Starting Ranger...
I20260926 16:38:34.086184 10680 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:38:34.536407 10680 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:38:34.536528 10680 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-admin
2026-09-26 16:38:34,829 [I] DB FLAVOR :POSTGRES
2026-09-26 16:38:34,830 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:38:34,830 [I] Checking connection..
2026-09-26 16:38:34,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:38:35,068 [I] Checking connection passed.
2026-09-26 16:38:35,068 [I] --------- Verifying version history table ---------
2026-09-26 16:38:35,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:38:35,324 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:38:35,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:38:35,555 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:38:35,556 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:38:35,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:38:35.804 UTC [21607] WARNING: there is no transaction in progress
2026-09-26 16:38:35,809 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:38:35,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:38:36,064 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:38:36,064 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:38:36,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:38:36,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:38:36,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-whjp.c.gcp-upstream.internal','N') ;"
2026-09-26 16:38:36,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:38:37,059 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:38:37,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:38:37,322 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:38:37,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:38:37,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:38:37,800 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:38:37,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:38:38.728 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.743 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.758 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.773 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.789 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.807 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.862 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.868 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.882 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.891 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.901 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.914 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.924 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.930 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.937 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:38.944 UTC [21715] WARNING: there is no transaction in progress
2026-09-26 16:38:39,527 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:38:39,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:38:39,753 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:38:39,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:38:40,005 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:38:40,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:38:40,262 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:38:40,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:38:40,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:38:40,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-whjp.c.gcp-upstream.internal';"
2026-09-26 16:38:40,976 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:38:40,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:38:41,213 [I] DB_PATCHES have already been applied
I20260926 16:38:41.218760 10680 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiK5LtG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:38:41.218864 10680 mini_ranger.cc:193] Using host: 127.10.110.20
I20260926 16:38:41.232764 10680 mini_ranger.cc:227] Ranger admin URL: http://127.10.110.20:54955
Sep 26, 2026 4:38:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:38:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:38:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:38:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:38:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:38:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:38:42.272336 10680 mini_ranger.cc:160] Time spent starting Ranger: real 8.211s user 0.001s sys 0.037s
Sep 26, 2026 4:39:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:39:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 4:39:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:39:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:39:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:39:04.167927 10680 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/rangerkms_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/rangerkms_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.110.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.110.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:39:04 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440744, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:39:04.325737 10680 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:39:04.344034 10680 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:39:05.136672 10680 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:39:05.136816 10680 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms
I20260926 16:39:05.136945 10680 mini_ranger_kms.cc:227] Using postgres at 127.10.110.20:49095
2026-09-26 16:39:05,513 [I] DB FLAVOR :POSTGRES
2026-09-26 16:39:05,514 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:39:05,514 [I] Checking connection
2026-09-26 16:39:05,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:39:05,755 [I] connection success
2026-09-26 16:39:05,756 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:39:05,756 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:39:05,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:39:05,985 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:39:05,985 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:39:05,985 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:39:05,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:49095/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:39:06,243 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:39:06.709932 10680 mini_ranger_kms.cc:310] Created kms service
I20260926 16:39:06.878911 10680 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:39:06.919813 10680 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:39:06 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440744, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:07.333045 10680 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:39:07.333302 10680 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:39:07.333345 10680 mini_ranger_kms.cc:240] Using host: 127.10.110.20
I20260926 16:39:07.349984 10680 mini_ranger_kms.cc:276] Ranger KMS PID: 21895
I20260926 16:39:07.350189 10680 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.110.20:53213
Sep 26, 2026 4:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:39:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 16:39:09.480751 10680 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.155s user 0.120s sys 0.273s
I20260926 16:39:09.481014 10680 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:39:09.481241 10680 mini_ranger_kms.cc:401] 127.10.110.20:53213/kms/v1/keys
Sep 26 16:39:09 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.110.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 16:39:09 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.110.20@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
Sep 26 16:39:10 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.110.20@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
Sep 26 16:39:12 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:39:12.474679 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:41889
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:41889
--ranger_config_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:53213/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:39:12.801791 21946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:39:12.802057 21946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:39:12.802172 21946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:39:12.809381 21946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:39:12.809491 21946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:39:12.809547 21946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:39:12.809621 21946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:39:12.809680 21946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:39:12.819633 21946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.110.20:53213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:41889
--ranger_config_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:39:12.821383 21946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:39:12.822659 21946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:39:12.835460 21952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:12.835925 21954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:12.836364 21951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:39:12.837363 21946 server_base.cc:1034] running on GCE node
I20260926 16:39:12.838117 21946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:39:12.839519 21946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:39:12.840713 21946 hybrid_clock.cc:648] HybridClock initialized: now 1790440752840641 us; error 65 us; skew 500 ppm
I20260926 16:39:12.841018 21946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:39:12 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:39:12.844033 21946 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:39:12.845518 21946 webserver.cc:460] Webserver started at http://127.10.110.62:39353/ using document root <none> and password file <none>
I20260926 16:39:12.846127 21946 fs_manager.cc:359] Metadata directory not provided
I20260926 16:39:12.846215 21946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:39:12.846527 21946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:39:12 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:12.895617 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "a68f7fa692504a68b39feda6456527f4"
format_stamp: "Formatted at 2026-09-26 16:39:12 on dist-test-slave-whjp"
server_key: "f8267e939f96f1e1f3c68266a522457a"
server_key_iv: "65d626b71c2f4867de41c7cafe35ded0"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:12.896486 21946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "a68f7fa692504a68b39feda6456527f4"
format_stamp: "Formatted at 2026-09-26 16:39:12 on dist-test-slave-whjp"
server_key: "f8267e939f96f1e1f3c68266a522457a"
server_key_iv: "65d626b71c2f4867de41c7cafe35ded0"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:12.901786 21946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:39:12.917796 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:12.919227 21946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:39:12.919397 21946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "a68f7fa692504a68b39feda6456527f4"
format_stamp: "Formatted at 2026-09-26 16:39:12 on dist-test-slave-whjp"
server_key: "f8267e939f96f1e1f3c68266a522457a"
server_key_iv: "65d626b71c2f4867de41c7cafe35ded0"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:12.919617 21946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:39:12.939373 21946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:39:12.943604 21946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:39:12.943918 21946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:39:12.962669 21946 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:41889
I20260926 16:39:12.962685 22012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:41889 every 8 connection(s)
I20260926 16:39:12.964072 21946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:39:12.968171 22013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:39:12.968752 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 21946
I20260926 16:39:12.968927 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:39:12.969682 10680 external_mini_cluster.cc:1428] Setting key d20c54b9b5bcdbcbd9eca84c8f086f50
I20260926 16:39:12.982847 22013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4: Bootstrap starting.
I20260926 16:39:12.987349 22013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:39:12.989339 22013 log.cc:824] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:39:12.993032 22013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4: No bootstrap required, opened a new log
Sep 26 16:39:12 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:39:13.004603 22013 raft_consensus.cc:348] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68f7fa692504a68b39feda6456527f4" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41889 } }
I20260926 16:39:13.005087 22013 raft_consensus.cc:374] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:39:13.005193 22013 raft_consensus.cc:729] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68f7fa692504a68b39feda6456527f4, State: Initialized, Role: FOLLOWER
I20260926 16:39:13.005863 22013 consensus_queue.cc:260] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68f7fa692504a68b39feda6456527f4" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41889 } }
I20260926 16:39:13.006067 22016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:12.971770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.10.110.62:41889)
0926 16:39:12.986650 (+ 14880us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:12.986666 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:12.986717 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:39:12.989484 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:12.989526 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:12.990118 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:12.990565 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:12.991957 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:12.993313 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:12.994564 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:12.995245 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:12.995411 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:12.998955 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:12.998964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:12.998981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:12.999028 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:13.001849 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:13.002505 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:13.002511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:13.002521 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:13.002669 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:13.003030 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:13.003034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:13.003036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:13.003626 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:13.003932 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:13.004357 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 16:39:13.004797 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14634,"thread_start_us":14405,"threads_started":1}
I20260926 16:39:13.006146 22013 raft_consensus.cc:388] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:39:13.006312 22013 raft_consensus.cc:482] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:39:13.006517 22013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:39:13.009785 22013 raft_consensus.cc:504] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68f7fa692504a68b39feda6456527f4" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41889 } }
I20260926 16:39:13.010375 22013 leader_election.cc:302] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a68f7fa692504a68b39feda6456527f4; no voters:
I20260926 16:39:13.010974 22013 leader_election.cc:288] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:39:13.011068 22018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:39:13.012181 22018 raft_consensus.cc:686] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [term 1 LEADER]: Becoming Leader. State: Replica: a68f7fa692504a68b39feda6456527f4, State: Running, Role: LEADER
I20260926 16:39:13.012730 22018 consensus_queue.cc:237] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68f7fa692504a68b39feda6456527f4" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41889 } }
I20260926 16:39:13.013056 22013 sys_catalog.cc:564] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:39:13.015916 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a68f7fa692504a68b39feda6456527f4. Latest consensus state: current_term: 1 leader_uuid: "a68f7fa692504a68b39feda6456527f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68f7fa692504a68b39feda6456527f4" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41889 } } }
I20260926 16:39:13.016394 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:39:13.016857 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a68f7fa692504a68b39feda6456527f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a68f7fa692504a68b39feda6456527f4" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 41889 } } }
I20260926 16:39:13.017123 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:39:13.017742 22023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:39:13.035017 22023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:39:13.046685 22023 catalog_manager.cc:1347] Generated new cluster ID: 047e4549684a40d88b568c8c90eb895f
I20260926 16:39:13.046810 22023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:39:13.067286 22023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:39:13.069169 22023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:39:13.080255 22023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4: Generated new TSK 0
I20260926 16:39:13.081323 22023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:39:13.084426 22013 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:39:13.955423 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:53213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41889
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:39:14 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440754, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:39:14.439280 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:39:14.439725 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:39:14.439970 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:39:14.451889 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:39:14.452124 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:39:14.452350 22058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:39:14.469300 22058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.110.20:53213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:39:14.470758 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:39:14.472638 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:39:14.490594 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:14.491142 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:39:14.504806 22058 server_base.cc:1034] running on GCE node
W20260926 16:39:14.505587 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:39:14.506289 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:39:14.507355 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:39:14.508625 22058 hybrid_clock.cc:648] HybridClock initialized: now 1790440754508523 us; error 94 us; skew 500 ppm
I20260926 16:39:14.509078 22058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:39:14 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440754, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:39:14.514149 22058 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:39:14.519289 22058 webserver.cc:460] Webserver started at http://127.10.110.1:39367/ using document root <none> and password file <none>
I20260926 16:39:14.520133 22058 fs_manager.cc:359] Metadata directory not provided
I20260926 16:39:14.520264 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:39:14.520576 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:39:14 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440754, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:14.537958 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "0b24fa1569e2437b86db4a2ef2281eda"
format_stamp: "Formatted at 2026-09-26 16:39:14 on dist-test-slave-whjp"
server_key: "d93441de95cd0d88a0b9da9487a55fad"
server_key_iv: "cc68bc26b7c2e3501729c3a5ce35758b"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:14.538797 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "0b24fa1569e2437b86db4a2ef2281eda"
format_stamp: "Formatted at 2026-09-26 16:39:14 on dist-test-slave-whjp"
server_key: "d93441de95cd0d88a0b9da9487a55fad"
server_key_iv: "cc68bc26b7c2e3501729c3a5ce35758b"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:14.545841 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:39:14.580230 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:14.582361 22058 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260926 16:39:14.582665 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "0b24fa1569e2437b86db4a2ef2281eda"
format_stamp: "Formatted at 2026-09-26 16:39:14 on dist-test-slave-whjp"
server_key: "d93441de95cd0d88a0b9da9487a55fad"
server_key_iv: "cc68bc26b7c2e3501729c3a5ce35758b"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:14.583015 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:39:14.637148 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:39:14.642998 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:39:14.643327 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:39:14.648780 22058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:39:14.652671 22058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:39:14.652839 22058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:39:14.653038 22058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:39:14.653111 22058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:39:14 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440754, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:14.889761 22058 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:40533
I20260926 16:39:14.889803 22190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:40533 every 8 connection(s)
I20260926 16:39:14.891374 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:39:14.894901 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 22058
I20260926 16:39:14.895087 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:39:14.896041 10680 external_mini_cluster.cc:1428] Setting key f31e6bf4bfe727a28a93f0bead8f7587
Sep 26 16:39:14 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440754, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:39:14.941435 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:14.894399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:39717 (local address 127.10.110.62:41889)
0926 16:39:14.917369 (+ 22970us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:14.917376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:14.917401 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:39:14.922453 (+ 5052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:14.922461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:14.922572 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:14.922652 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:14.925165 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:14.926317 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:14.928718 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:14.929326 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:14.929415 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:14.933731 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:14.933738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:14.933742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:14.933783 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:14.937754 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:14.938689 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:14.938696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:14.938699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:14.938837 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:14.939328 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:14.939333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:14.939337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:14.939686 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:14.939805 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:14.940828 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 16:39:14.941057 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22835,"thread_start_us":22706,"threads_started":1}
I20260926 16:39:14.943598 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:14.895022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41889 (local address 127.10.110.1:39717)
0926 16:39:14.916454 (+ 21432us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:14.916524 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:14.922050 (+ 5526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:14.923338 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:14.923353 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:14.924006 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:39:14.924952 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:14.924976 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:14.926528 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:14.926534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:14.928554 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:14.928568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:14.929763 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:14.929769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:14.930053 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:14.930844 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:39:14.930875 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:39:14.933469 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:39:14.937932 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:39:14.937942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:39:14.937962 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:39:14.938551 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:39:14.938972 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:39:14.938976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:39:14.938980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:39:14.939209 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:39:14.939917 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:39:14.939926 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:39:14.940367 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 16:39:14.941724 (+ 1357us) client_negotiation.cc:240] Negotiation successful
0926 16:39:14.942265 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21121,"thread_start_us":20884,"threads_started":1}
I20260926 16:39:14.945451 22191 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41889
I20260926 16:39:14.945803 22191 heartbeater.cc:464] Registering TS with master...
I20260926 16:39:14.946646 22191 heartbeater.cc:511] Master 127.10.110.62:41889 requested a full tablet report, sending...
I20260926 16:39:14.949119 21978 ts_manager.cc:194] Registered new tserver with Master: 0b24fa1569e2437b86db4a2ef2281eda (127.10.110.1:40533)
I20260926 16:39:14.951157 21978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:39717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:39:14.970994 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:53213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41889
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:39:15 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440754, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
W20260926 16:39:15.401943 22197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:39:15.402237 22197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:39:15.402356 22197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:39:15.409997 22197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:39:15.410121 22197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:39:15.410288 22197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:39:15.420409 22197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.110.20:53213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:39:15.421684 22197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:39:15.423014 22197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:39:15.435910 22214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:15.435997 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:15.436560 22211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:39:15.437144 22197 server_base.cc:1034] running on GCE node
I20260926 16:39:15.437887 22197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:39:15.439090 22197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:39:15.440296 22197 hybrid_clock.cc:648] HybridClock initialized: now 1790440755440236 us; error 54 us; skew 500 ppm
I20260926 16:39:15.440613 22197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:39:15 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440755, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:39:15.444011 22197 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:39:15.445418 22197 webserver.cc:460] Webserver started at http://127.10.110.2:38801/ using document root <none> and password file <none>
I20260926 16:39:15.446091 22197 fs_manager.cc:359] Metadata directory not provided
I20260926 16:39:15.446187 22197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:39:15.446448 22197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:39:15 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440755, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:15.460232 22197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "1602eb57ff5b47efab78327aac02d2fd"
format_stamp: "Formatted at 2026-09-26 16:39:15 on dist-test-slave-whjp"
server_key: "28ed2b5dac2baae1dd712e5ed882eb77"
server_key_iv: "d4268e73a44299378160954fafac788a"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:15.461002 22197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "1602eb57ff5b47efab78327aac02d2fd"
format_stamp: "Formatted at 2026-09-26 16:39:15 on dist-test-slave-whjp"
server_key: "28ed2b5dac2baae1dd712e5ed882eb77"
server_key_iv: "d4268e73a44299378160954fafac788a"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:15.466091 22197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:39:15.481896 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:15.483403 22197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:39:15.483546 22197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "1602eb57ff5b47efab78327aac02d2fd"
format_stamp: "Formatted at 2026-09-26 16:39:15 on dist-test-slave-whjp"
server_key: "28ed2b5dac2baae1dd712e5ed882eb77"
server_key_iv: "d4268e73a44299378160954fafac788a"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:15.483744 22197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:39:15.502271 22197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:39:15.505920 22197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:39:15.506214 22197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:39:15.507123 22197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:39:15.508472 22197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:39:15.508541 22197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:15.508641 22197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:39:15.508688 22197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:15.543292 22197 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:45817
I20260926 16:39:15.543366 22333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:45817 every 8 connection(s)
I20260926 16:39:15.544601 22197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:39:15.553368 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 22197
I20260926 16:39:15.553546 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:39:15.554083 10680 external_mini_cluster.cc:1428] Setting key 02c70177860180cbf75b0474f2a8c15d
Sep 26 16:39:15 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440755, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:39:15.567286 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:15.547106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:49723 (local address 127.10.110.62:41889)
0926 16:39:15.547494 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:15.547499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:15.548709 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 16:39:15.550472 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:15.550478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:15.550568 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:15.550662 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:15.552143 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:15.553286 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:15.555371 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:15.555791 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:15.555926 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:15.560211 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:15.560217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:15.560221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:15.560261 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:15.564005 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:15.564731 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:15.564737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:15.564741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:15.564858 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:15.565318 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:15.565323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:15.565327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:15.565693 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:15.565838 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:15.566666 (+ 828us) server_negotiation.cc:299] Negotiation successful
0926 16:39:15.566901 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":196,"threads_started":1}
I20260926 16:39:15.568441 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:15.547765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41889 (local address 127.10.110.2:49723)
0926 16:39:15.548542 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:15.548599 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:15.550218 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:15.550903 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:15.550912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:15.551318 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:39:15.551934 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:15.551958 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:15.553478 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:15.553484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:15.555245 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:15.555258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:15.556170 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:15.556177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:15.556449 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:15.557188 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:39:15.557214 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:39:15.559947 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:39:15.564149 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:39:15.564158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:39:15.564177 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:39:15.564609 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:39:15.564962 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:39:15.564967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:39:15.564971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:39:15.565187 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:39:15.565822 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:39:15.565831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:39:15.566243 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 16:39:15.566522 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 16:39:15.567048 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":310,"threads_started":1}
I20260926 16:39:15.570380 22334 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41889
I20260926 16:39:15.570725 22334 heartbeater.cc:464] Registering TS with master...
I20260926 16:39:15.571575 22334 heartbeater.cc:511] Master 127.10.110.62:41889 requested a full tablet report, sending...
I20260926 16:39:15.573391 21978 ts_manager.cc:194] Registered new tserver with Master: 1602eb57ff5b47efab78327aac02d2fd (127.10.110.2:45817)
I20260926 16:39:15.574579 21978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:49723
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:39:15.602766 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:53213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41889
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:40981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:39:15.928650 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:39:15.928911 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:39:15.929009 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:39:15.936005 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:39:15.936112 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:39:15.936228 22340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:39:15.946118 22340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.110.20:53213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:39:15.947294 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:39:15.948542 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:39:15.954852 22191 heartbeater.cc:503] Master 127.10.110.62:41889 was elected leader, sending a full tablet report...
W20260926 16:39:15.961184 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:15.961148 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:39:15.961223 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:39:15.962373 22340 server_base.cc:1034] running on GCE node
I20260926 16:39:15.962929 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:39:15.963809 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:39:15.965034 22340 hybrid_clock.cc:648] HybridClock initialized: now 1790440755964964 us; error 51 us; skew 500 ppm
I20260926 16:39:15.965319 22340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:39:15 dist-test-slave-whjp krb5kdc[21502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440755, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:39:15.968314 22340 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:39:15.969676 22340 webserver.cc:460] Webserver started at http://127.10.110.3:42565/ using document root <none> and password file <none>
I20260926 16:39:15.970315 22340 fs_manager.cc:359] Metadata directory not provided
I20260926 16:39:15.970424 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:39:15.970691 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:39:15 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440755, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:15.986696 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "4762dce63d3f4dde9c2668a8af137e81"
format_stamp: "Formatted at 2026-09-26 16:39:15 on dist-test-slave-whjp"
server_key: "d39b8078026177c1d37bf6fae09a952b"
server_key_iv: "80bc10c742c0e608a74b1a130c699c98"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:15.987552 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "4762dce63d3f4dde9c2668a8af137e81"
format_stamp: "Formatted at 2026-09-26 16:39:15 on dist-test-slave-whjp"
server_key: "d39b8078026177c1d37bf6fae09a952b"
server_key_iv: "80bc10c742c0e608a74b1a130c699c98"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:15.993010 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:39:16.008080 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:16.009837 22340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:39:16.010047 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "4762dce63d3f4dde9c2668a8af137e81"
format_stamp: "Formatted at 2026-09-26 16:39:15 on dist-test-slave-whjp"
server_key: "d39b8078026177c1d37bf6fae09a952b"
server_key_iv: "80bc10c742c0e608a74b1a130c699c98"
server_key_version: "kuduclusterkey@0"
I20260926 16:39:16.010336 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:39:16.029559 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:39:16.033311 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:39:16.033571 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:39:16.034430 22340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:39:16.035660 22340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:39:16.035727 22340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:39:16.035818 22340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:39:16.035899 22340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 16:39:16.067239 22340 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:40027
I20260926 16:39:16.067273 22467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:40027 every 8 connection(s)
I20260926 16:39:16.068599 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:39:16.073596 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 22340
I20260926 16:39:16.073776 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:39:16.074312 10680 external_mini_cluster.cc:1428] Setting key f9b1aa52284b5debf951dcd0cab0bf01
Sep 26 16:39:16 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440755, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:39:16.088840 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.071666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:44271 (local address 127.10.110.62:41889)
0926 16:39:16.072425 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.072430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.073949 (+ 1519us) server_negotiation.cc:407] Connection header received
0926 16:39:16.075927 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.075933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.076035 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.076135 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:16.077721 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.078410 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.079942 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.080374 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.080479 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.083544 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:16.083549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:16.083552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:16.083583 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:16.085933 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.086608 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.086612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.086615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.086730 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.087157 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.087162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.087165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.087419 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:16.087540 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.088256 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.088479 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":534,"threads_started":1}
I20260926 16:39:16.089923 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.072636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:41889 (local address 127.10.110.3:44271)
0926 16:39:16.073783 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:16.073830 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:16.075658 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:16.076392 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:16.076404 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:16.076957 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:39:16.077571 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.077588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.078550 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.078554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.079777 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.079795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.080576 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.080583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.080800 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.081380 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:39:16.081402 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:39:16.083337 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:39:16.086060 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:39:16.086068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:39:16.086086 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:39:16.086494 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:39:16.086826 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:39:16.086831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:39:16.086835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:39:16.087059 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:39:16.087532 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:39:16.087540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:39:16.087899 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 16:39:16.088182 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 16:39:16.088680 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":724,"threads_started":1}
I20260926 16:39:16.091110 22468 heartbeater.cc:347] Connected to a master server at 127.10.110.62:41889
I20260926 16:39:16.091380 22468 heartbeater.cc:464] Registering TS with master...
I20260926 16:39:16.092067 22468 heartbeater.cc:511] Master 127.10.110.62:41889 requested a full tablet report, sending...
I20260926 16:39:16.093743 21978 ts_manager.cc:194] Registered new tserver with Master: 4762dce63d3f4dde9c2668a8af137e81 (127.10.110.3:40027)
I20260926 16:39:16.094515 21978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:44271
I20260926 16:39:16.102473 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:39:16.118414 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.106792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58890 (local address 127.10.110.62:41889)
0926 16:39:16.106950 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.106954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.107292 (+ 338us) server_negotiation.cc:407] Connection header received
0926 16:39:16.107439 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.107443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.107518 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.107614 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:16.108942 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.109665 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.111029 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.111444 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.111546 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.113867 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:16.113874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:16.113879 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:16.113913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:16.116367 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.116975 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.116979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.116981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.117096 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.117361 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.117363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.117365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.117620 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:16.117790 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.118030 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.118190 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:39:16.122795 21978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:39:16.129805 21978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:39:16.163743 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.142509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.10.110.2:45817)
0926 16:39:16.153854 (+ 11345us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.153866 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.154573 (+ 707us) server_negotiation.cc:407] Connection header received
0926 16:39:16.155591 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.155600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.155898 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.156296 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:39:16.156886 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.158156 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.161310 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.162620 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.162742 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.162941 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.163242 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.163410 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11184,"thread_start_us":11059,"threads_started":1}
I20260926 16:39:16.163796 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.142104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:45817 (local address 127.0.0.1:42440)
0926 16:39:16.154408 (+ 12304us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:16.154459 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:16.154678 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:16.156318 (+ 1640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:16.156324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:16.156346 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:39:16.156667 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.156678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.158278 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.158283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.161145 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.161162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.162799 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.162805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.162984 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.163036 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 16:39:16.163189 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 16:39:16.163366 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12010,"mutex_wait_us":3,"thread_start_us":14910,"threads_started":1}
I20260926 16:39:16.164119 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.142313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:40533 (local address 127.0.0.1:39388)
0926 16:39:16.156051 (+ 13738us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:16.156072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:16.156181 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:16.156719 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:16.156724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:16.156746 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:39:16.157151 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.157164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.159125 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.159130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.161541 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.161550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.163087 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.163093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.163248 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.163277 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:39:16.163348 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:39:16.163442 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13628,"mutex_wait_us":1,"thread_start_us":11655,"threads_started":1}
I20260926 16:39:16.164281 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.142422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.10.110.1:40533)
0926 16:39:16.156168 (+ 13746us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.156177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.156206 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:39:16.156292 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.156299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.156507 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.156715 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:39:16.157363 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.158954 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.161676 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.162757 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.162890 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.163089 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.163804 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.163977 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13553,"thread_start_us":13574,"threads_started":1}
I20260926 16:39:16.164824 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.144798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:40027 (local address 127.0.0.1:55864)
0926 16:39:16.157290 (+ 12492us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:16.157310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:16.157415 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:16.158018 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:16.158023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:16.158046 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:39:16.158425 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.158438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.160077 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.160081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.162220 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.162235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.164097 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.164107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.164264 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.164300 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:39:16.164371 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:39:16.164485 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12374,"mutex_wait_us":1,"thread_start_us":11012,"threads_started":1}
I20260926 16:39:16.165619 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.145400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.10.110.3:40027)
0926 16:39:16.155628 (+ 10228us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.155638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.157537 (+ 1899us) server_negotiation.cc:407] Connection header received
0926 16:39:16.157635 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.157642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.157852 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.158021 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:39:16.158607 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.159920 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.162373 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.163687 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.163806 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.164781 (+ 975us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.165124 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.165277 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10053,"thread_start_us":9911,"threads_started":1}
I20260926 16:39:16.167747 22269 tablet_service.cc:1467] Processing CreateTablet for tablet 2bffaa4421514d7ead3e1f4629008f93 (DEFAULT_TABLE table=test-table [id=b71e36ebbe8d4e89bd77a8ebeec6901c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:39:16.167855 22126 tablet_service.cc:1467] Processing CreateTablet for tablet 2bffaa4421514d7ead3e1f4629008f93 (DEFAULT_TABLE table=test-table [id=b71e36ebbe8d4e89bd77a8ebeec6901c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:39:16.168975 22269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bffaa4421514d7ead3e1f4629008f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:39:16.169186 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bffaa4421514d7ead3e1f4629008f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:39:16.168915 22403 tablet_service.cc:1467] Processing CreateTablet for tablet 2bffaa4421514d7ead3e1f4629008f93 (DEFAULT_TABLE table=test-table [id=b71e36ebbe8d4e89bd77a8ebeec6901c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:39:16.170200 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bffaa4421514d7ead3e1f4629008f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:39:16.180035 22487 tablet_bootstrap.cc:492] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Bootstrap starting.
I20260926 16:39:16.182587 22488 tablet_bootstrap.cc:492] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Bootstrap starting.
I20260926 16:39:16.184109 22487 tablet_bootstrap.cc:654] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Neither blocks nor log segments found. Creating new log.
I20260926 16:39:16.184173 22489 tablet_bootstrap.cc:492] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Bootstrap starting.
I20260926 16:39:16.185554 22487 log.cc:824] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Log is configured to *not* fsync() on all Append() calls
I20260926 16:39:16.186378 22488 tablet_bootstrap.cc:654] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Neither blocks nor log segments found. Creating new log.
I20260926 16:39:16.186872 22489 tablet_bootstrap.cc:654] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Neither blocks nor log segments found. Creating new log.
I20260926 16:39:16.187460 22488 log.cc:824] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Log is configured to *not* fsync() on all Append() calls
I20260926 16:39:16.187737 22489 log.cc:824] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Log is configured to *not* fsync() on all Append() calls
I20260926 16:39:16.188689 22487 tablet_bootstrap.cc:492] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: No bootstrap required, opened a new log
I20260926 16:39:16.188944 22487 ts_tablet_manager.cc:1397] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 16:39:16.190230 22488 tablet_bootstrap.cc:492] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: No bootstrap required, opened a new log
I20260926 16:39:16.190479 22489 tablet_bootstrap.cc:492] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: No bootstrap required, opened a new log
I20260926 16:39:16.190472 22488 ts_tablet_manager.cc:1397] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 16:39:16.190696 22489 ts_tablet_manager.cc:1397] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 16:39:16.196537 22487 raft_consensus.cc:348] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.196889 22487 raft_consensus.cc:374] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:39:16.196974 22487 raft_consensus.cc:729] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1602eb57ff5b47efab78327aac02d2fd, State: Initialized, Role: FOLLOWER
I20260926 16:39:16.197517 22487 consensus_queue.cc:260] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.198521 22334 heartbeater.cc:503] Master 127.10.110.62:41889 was elected leader, sending a full tablet report...
I20260926 16:39:16.199146 22487 ts_tablet_manager.cc:1428] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260926 16:39:16.200366 22488 raft_consensus.cc:348] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.200789 22488 raft_consensus.cc:374] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:39:16.200888 22488 raft_consensus.cc:729] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b24fa1569e2437b86db4a2ef2281eda, State: Initialized, Role: FOLLOWER
I20260926 16:39:16.200793 22489 raft_consensus.cc:348] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.201174 22489 raft_consensus.cc:374] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:39:16.201274 22489 raft_consensus.cc:729] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4762dce63d3f4dde9c2668a8af137e81, State: Initialized, Role: FOLLOWER
I20260926 16:39:16.201567 22488 consensus_queue.cc:260] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.201802 22489 consensus_queue.cc:260] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.203473 22489 ts_tablet_manager.cc:1428] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260926 16:39:16.203541 22488 ts_tablet_manager.cc:1428] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260926 16:39:16.203907 22468 heartbeater.cc:503] Master 127.10.110.62:41889 was elected leader, sending a full tablet report...
I20260926 16:39:16.265805 22493 raft_consensus.cc:482] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:39:16.266451 22493 raft_consensus.cc:504] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.268111 22493 leader_election.cc:288] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b24fa1569e2437b86db4a2ef2281eda (127.10.110.1:40533), 4762dce63d3f4dde9c2668a8af137e81 (127.10.110.3:40027)
I20260926 16:39:16.274434 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.268501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:54207 (local address 127.10.110.1:40533)
0926 16:39:16.268730 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.268737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.269329 (+ 592us) server_negotiation.cc:407] Connection header received
0926 16:39:16.269506 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.269511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.269573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.269636 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:39:16.270210 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.270944 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.272804 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.273570 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.273724 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.273792 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.274090 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.274207 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:39:16.274448 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.268597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:55105 (local address 127.10.110.3:40027)
0926 16:39:16.268898 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.268903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.269382 (+ 479us) server_negotiation.cc:407] Connection header received
0926 16:39:16.269547 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.269550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.269608 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.269704 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:39:16.270274 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.270939 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.272755 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.273527 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.273651 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.273721 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.274104 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.274219 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260926 16:39:16.274554 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.268336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:40533 (local address 127.10.110.2:54207)
0926 16:39:16.269200 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:16.269220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:16.269367 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:16.269684 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:16.269689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:16.269719 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:39:16.270068 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.270076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.271061 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.271066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.272655 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.272667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.273794 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.273798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.273935 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.273965 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:39:16.274038 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 16:39:16.274157 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":61,"thread_start_us":477,"threads_started":1}
I20260926 16:39:16.274705 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.268523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:40027 (local address 127.10.110.2:55105)
0926 16:39:16.269271 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 16:39:16.269291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:39:16.269419 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:39:16.269728 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:39:16.269732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:39:16.269755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:39:16.270120 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.270129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.271060 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.271065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.272605 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:39:16.272617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.273749 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.273756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:39:16.273902 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.273971 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 16:39:16.274050 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:39:16.274166 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":80,"thread_start_us":514,"threads_started":1}
I20260926 16:39:16.275619 22146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bffaa4421514d7ead3e1f4629008f93" candidate_uuid: "1602eb57ff5b47efab78327aac02d2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b24fa1569e2437b86db4a2ef2281eda" is_pre_election: true
I20260926 16:39:16.276190 22146 raft_consensus.cc:2454] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1602eb57ff5b47efab78327aac02d2fd in term 0.
I20260926 16:39:16.276258 22423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bffaa4421514d7ead3e1f4629008f93" candidate_uuid: "1602eb57ff5b47efab78327aac02d2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4762dce63d3f4dde9c2668a8af137e81" is_pre_election: true
I20260926 16:39:16.276688 22423 raft_consensus.cc:2454] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1602eb57ff5b47efab78327aac02d2fd in term 0.
I20260926 16:39:16.276891 22225 leader_election.cc:302] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b24fa1569e2437b86db4a2ef2281eda, 1602eb57ff5b47efab78327aac02d2fd; no voters:
I20260926 16:39:16.277246 22493 raft_consensus.cc:2781] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:39:16.277365 22493 raft_consensus.cc:482] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:39:16.277467 22493 raft_consensus.cc:3037] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:39:16.279531 22493 raft_consensus.cc:504] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.280126 22493 leader_election.cc:288] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [CANDIDATE]: Term 1 election: Requested vote from peers 0b24fa1569e2437b86db4a2ef2281eda (127.10.110.1:40533), 4762dce63d3f4dde9c2668a8af137e81 (127.10.110.3:40027)
I20260926 16:39:16.280611 22423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bffaa4421514d7ead3e1f4629008f93" candidate_uuid: "1602eb57ff5b47efab78327aac02d2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4762dce63d3f4dde9c2668a8af137e81"
I20260926 16:39:16.280614 22146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bffaa4421514d7ead3e1f4629008f93" candidate_uuid: "1602eb57ff5b47efab78327aac02d2fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b24fa1569e2437b86db4a2ef2281eda"
I20260926 16:39:16.280779 22423 raft_consensus.cc:3037] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:39:16.280777 22146 raft_consensus.cc:3037] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:39:16.282840 22423 raft_consensus.cc:2454] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1602eb57ff5b47efab78327aac02d2fd in term 1.
I20260926 16:39:16.282842 22146 raft_consensus.cc:2454] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1602eb57ff5b47efab78327aac02d2fd in term 1.
I20260926 16:39:16.283234 22223 leader_election.cc:302] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1602eb57ff5b47efab78327aac02d2fd, 4762dce63d3f4dde9c2668a8af137e81; no voters:
I20260926 16:39:16.283519 22493 raft_consensus.cc:2781] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:39:16.283921 22493 raft_consensus.cc:686] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 1 LEADER]: Becoming Leader. State: Replica: 1602eb57ff5b47efab78327aac02d2fd, State: Running, Role: LEADER
I20260926 16:39:16.284363 22493 consensus_queue.cc:237] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } }
I20260926 16:39:16.288949 21976 catalog_manager.cc:5495] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1602eb57ff5b47efab78327aac02d2fd (127.10.110.2). New cstate: current_term: 1 leader_uuid: "1602eb57ff5b47efab78327aac02d2fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1602eb57ff5b47efab78327aac02d2fd" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 } health_report { overall_health: UNKNOWN } } }
W20260926 16:39:16.296499 22335 tablet.cc:2369] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:39:16.306609 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.301425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.10.110.2:45817)
0926 16:39:16.301607 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.301612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.301630 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:39:16.301805 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.301809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.301893 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.302017 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:39:16.302383 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.303105 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.304569 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.304990 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.305114 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.305563 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:39:16.306042 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:39:16.306183 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.306277 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.306356 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:39:16.314062 22423 raft_consensus.cc:1261] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 1 FOLLOWER]: Refusing update from remote peer 1602eb57ff5b47efab78327aac02d2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:39:16.314117 22146 raft_consensus.cc:1261] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 1 FOLLOWER]: Refusing update from remote peer 1602eb57ff5b47efab78327aac02d2fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:39:16.314953 22498 consensus_queue.cc:1035] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b24fa1569e2437b86db4a2ef2281eda" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:39:16.315281 22493 consensus_queue.cc:1035] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4762dce63d3f4dde9c2668a8af137e81" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 40027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:39:16.320052 22469 tablet.cc:2369] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:39:16.331277 22502 mvcc.cc:205] Tried to move back new op lower bound from 7333645337853362176 to 7333645337744199680. Current Snapshot: MvccSnapshot[applied={T|T < 7333645337853362176}]
I20260926 16:39:16.353399 21976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:39:16.353693 21976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:39:16.357124 21976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a68f7fa692504a68b39feda6456527f4: Sending DeleteTablet for 3 replicas of tablet 2bffaa4421514d7ead3e1f4629008f93
I20260926 16:39:16.358153 22269 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bffaa4421514d7ead3e1f4629008f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:39:16 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42440
I20260926 16:39:16.358328 22403 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bffaa4421514d7ead3e1f4629008f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:39:16 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:55864
I20260926 16:39:16.359244 22126 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bffaa4421514d7ead3e1f4629008f93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:39:16 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:39388
I20260926 16:39:16.359999 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 22058
I20260926 16:39:16.378980 22512 tablet_replica.cc:330] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: stopping tablet replica
I20260926 16:39:16.379613 22512 raft_consensus.cc:2229] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:39:16.380276 22513 tablet_replica.cc:330] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: stopping tablet replica
I20260926 16:39:16.380596 22513 raft_consensus.cc:2229] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:39:16.381004 22513 raft_consensus.cc:2258] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:39:16.381691 22512 raft_consensus.cc:2258] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:39:16.383510 22513 ts_tablet_manager.cc:1905] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:39:16.383625 22512 ts_tablet_manager.cc:1905] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:39:16 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:39:16.388489 22511 tablet_replica.cc:330] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: stopping tablet replica
I20260926 16:39:16.388957 22511 raft_consensus.cc:2229] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:39:16.389712 22511 raft_consensus.cc:2258] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:39:16.390914 22513 ts_tablet_manager.cc:1918] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:39:16.391057 22513 log.cc:1196] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/2bffaa4421514d7ead3e1f4629008f93
I20260926 16:39:16.391554 22513 ts_tablet_manager.cc:1939] T 2bffaa4421514d7ead3e1f4629008f93 P 0b24fa1569e2437b86db4a2ef2281eda: Deleting consensus metadata
I20260926 16:39:16.392560 22512 ts_tablet_manager.cc:1918] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:39:16.392858 22512 log.cc:1196] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/2bffaa4421514d7ead3e1f4629008f93
I20260926 16:39:16.393332 22512 ts_tablet_manager.cc:1939] T 2bffaa4421514d7ead3e1f4629008f93 P 4762dce63d3f4dde9c2668a8af137e81: Deleting consensus metadata
I20260926 16:39:16.393461 21965 catalog_manager.cc:4841] TS 0b24fa1569e2437b86db4a2ef2281eda (127.10.110.1:40533): tablet 2bffaa4421514d7ead3e1f4629008f93 (table test-table [id=b71e36ebbe8d4e89bd77a8ebeec6901c]) successfully deleted
I20260926 16:39:16.394369 22511 ts_tablet_manager.cc:1905] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:39:16.394757 21963 catalog_manager.cc:4841] TS 4762dce63d3f4dde9c2668a8af137e81 (127.10.110.3:40027): tablet 2bffaa4421514d7ead3e1f4629008f93 (table test-table [id=b71e36ebbe8d4e89bd77a8ebeec6901c]) successfully deleted
I20260926 16:39:16.395588 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.361415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.10.110.1:40533)
0926 16:39:16.361583 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.361589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.371314 (+ 9725us) server_negotiation.cc:407] Connection header received
0926 16:39:16.371790 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.371797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.371904 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.371983 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:16.373760 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.374902 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.376892 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.377595 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.377696 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.387232 (+ 9536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:16.387250 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:16.387268 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:16.387320 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:16.391438 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.392677 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.392686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.392689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.392999 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.393563 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.393568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.393571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.394011 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:16.394294 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.394966 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.395207 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:39:16.396250 22186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:39:16.400941 22511 ts_tablet_manager.cc:1918] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:39:16.401078 22511 log.cc:1196] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/2bffaa4421514d7ead3e1f4629008f93
I20260926 16:39:16.401460 22511 ts_tablet_manager.cc:1939] T 2bffaa4421514d7ead3e1f4629008f93 P 1602eb57ff5b47efab78327aac02d2fd: Deleting consensus metadata
I20260926 16:39:16.402742 21962 catalog_manager.cc:4841] TS 1602eb57ff5b47efab78327aac02d2fd (127.10.110.2:45817): tablet 2bffaa4421514d7ead3e1f4629008f93 (table test-table [id=b71e36ebbe8d4e89bd77a8ebeec6901c]) successfully deleted
I20260926 16:39:16.534070 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 22058
I20260926 16:39:16.550799 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 22197
Sep 26 16:39:16 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:39:16.569802 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.551577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.10.110.2:45817)
0926 16:39:16.551756 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.551762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.551899 (+ 137us) server_negotiation.cc:407] Connection header received
0926 16:39:16.552151 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.552157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.552253 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.552363 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:16.553903 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.555141 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.556965 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.557630 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.557760 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.562801 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:16.562815 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:16.562838 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:16.562886 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:16.566840 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.567672 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.567681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.567685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.567876 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.568246 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.568251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.568254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.568687 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:16.569001 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.569230 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.569432 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:39:16.570492 22329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:39:16.683269 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 22197
I20260926 16:39:16.697218 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 22340
Sep 26 16:39:16 dist-test-slave-whjp krb5kdc[21502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:39:16.711719 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:39:16.698009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.10.110.3:40027)
0926 16:39:16.698222 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:39:16.698228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:39:16.698249 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:39:16.698396 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:39:16.698401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:39:16.698475 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:39:16.698559 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:39:16.699887 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.700600 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.701827 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:39:16.702214 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:39:16.702320 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:39:16.706439 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:39:16.706452 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:39:16.706467 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:39:16.706501 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:39:16.709096 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.709791 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.709799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.709801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.709923 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:39:16.710304 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:39:16.710308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:39:16.710310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:39:16.710648 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:39:16.710902 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 16:39:16.711198 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 16:39:16.711398 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:39:16.712474 22463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:39:16.799675 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 22340
I20260926 16:39:16.812500 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 21946
I20260926 16:39:16.813612 22008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:39:16.922676 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 21946
I20260926 16:39:16.943958 10680 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:39:17.279379 10680 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:39:17.279659 10680 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:39:17.614612 10680 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:39:17.614 UTC [21532] LOG: received smart shutdown request
2026-09-26 16:39:17.617 UTC [21532] LOG: background worker "logical replication launcher" (PID 21541) exited with exit code 1
2026-09-26 16:39:17.618 UTC [21536] LOG: shutting down
2026-09-26 16:39:17.767 UTC [21532] LOG: database system is shut down
2026-09-26T16:39:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48423 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:39:17 dist-test-slave-whjp krb5kdc[22527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:39:17 dist-test-slave-whjp krb5kdc[22527](info): set up 2 sockets
Sep 26 16:39:17 dist-test-slave-whjp krb5kdc[22527](info): commencing operation
krb5kdc: starting...
W20260926 16:39:18.896808 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:39:19 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:39:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/rangeradmin_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/rangeradmin_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/rangerlookup_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/rangerlookup_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
I20260926 16:39:19.186973 10680 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiK5LtG/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/postgres -l logfile start
2026-09-26 16:39:21.677 UTC [22557] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:39:21.677 UTC [22557] LOG: listening on IPv4 address "127.10.110.20", port 60349
2026-09-26 16:39:21.680 UTC [22557] LOG: listening on Unix socket "/tmp/.s.PGSQL.60349"
2026-09-26 16:39:21.700 UTC [22560] LOG: database system was shut down at 2026-09-26 16:39:20 UTC
2026-09-26 16:39:21.704 UTC [22557] LOG: database system is ready to accept connections
I20260926 16:39:22.604491 10680 mini_postgres.cc:96] Postgres bound to 60349
2026-09-26 16:39:22.623 UTC [22568] FATAL: database "slave" does not exist
127.10.110.20:60349 - accepting connections
I20260926 16:39:22.624249 10680 mini_ranger.cc:161] Starting Ranger...
I20260926 16:39:22.648777 10680 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:39:23.121914 10680 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:39:23.122011 10680 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-admin
2026-09-26 16:39:23,409 [I] DB FLAVOR :POSTGRES
2026-09-26 16:39:23,409 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:39:23,409 [I] Checking connection..
2026-09-26 16:39:23,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:39:23,650 [I] Checking connection passed.
2026-09-26 16:39:23,651 [I] --------- Verifying version history table ---------
2026-09-26 16:39:23,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:39:23,900 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:39:23,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:39:24,127 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:39:24,128 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:39:24,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:39:24.381 UTC [22632] WARNING: there is no transaction in progress
2026-09-26 16:39:24,387 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:39:24,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:39:24,628 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:39:24,628 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:39:24,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:39:24,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:39:25,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-whjp.c.gcp-upstream.internal','N') ;"
2026-09-26 16:39:25,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:39:25,587 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:39:25,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:39:25,841 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:39:25,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:39:26,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:39:26,316 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:39:26,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:39:27.242 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.259 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.277 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.293 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.310 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.328 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.391 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.398 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.412 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.421 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.431 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.445 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.454 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.461 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.468 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:27.475 UTC [22740] WARNING: there is no transaction in progress
2026-09-26 16:39:28,130 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:39:28,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:39:28,373 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:39:28,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:39:28,616 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:39:28,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:39:28,864 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:39:28,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:39:29,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:39:29,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-whjp.c.gcp-upstream.internal';"
2026-09-26 16:39:29,608 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:39:29,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:39:29,846 [I] DB_PATCHES have already been applied
I20260926 16:39:29.852185 10680 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiK5LtG/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:39:29.852272 10680 mini_ranger.cc:193] Using host: 127.10.110.20
I20260926 16:39:29.865147 10680 mini_ranger.cc:227] Ranger admin URL: http://127.10.110.20:48013
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:39:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:39:30.883181 10680 mini_ranger.cc:160] Time spent starting Ranger: real 8.259s user 0.004s sys 0.033s
Sep 26, 2026 4:39:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:39:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:39:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:39:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:39:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:39:52.336719 10680 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.110.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.110.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/rangerkms_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.110.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/rangerkms_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.110.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.110.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.110.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/HTTP_127.10.110.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:39:52 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440792, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:39:52.489862 10680 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:39:52.509536 10680 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:39:53.396672 10680 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:39:53.396766 10680 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms
I20260926 16:39:53.396797 10680 mini_ranger_kms.cc:227] Using postgres at 127.10.110.20:60349
2026-09-26 16:39:53,729 [I] DB FLAVOR :POSTGRES
2026-09-26 16:39:53,730 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:39:53,730 [I] Checking connection
2026-09-26 16:39:53,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:39:53,955 [I] connection success
2026-09-26 16:39:53,956 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:39:53,956 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:39:53,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:39:54,184 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:39:54,185 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:39:54,185 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:39:54,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.110.20:60349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:39:54,432 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:39:54.893297 10680 mini_ranger_kms.cc:310] Created kms service
I20260926 16:39:55.081427 10680 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:39:55.125348 10680 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:39:55 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440792, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:39:55.535159 10680 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:39:55.535483 10680 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiK5LtG/build/asan/bin/postgresql.jar:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiK5LtG/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:39:55.535542 10680 mini_ranger_kms.cc:240] Using host: 127.10.110.20
I20260926 16:39:55.551528 10680 mini_ranger_kms.cc:276] Ranger KMS PID: 22920
I20260926 16:39:55.551671 10680 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.110.20:50985
Sep 26, 2026 4:39:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:39:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:39:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:39:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:39:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:39:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 16:39:57.640476 10680 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.151s user 0.087s sys 0.265s
I20260926 16:39:57.640724 10680 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:39:57.640889 10680 mini_ranger_kms.cc:401] 127.10.110.20:50985/kms/v1/keys
Sep 26 16:39:57 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440797, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.110.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 16:39:58 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440797, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.110.20@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
Sep 26 16:39:58 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440797, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.110.20@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
Sep 26 16:40:00 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:40:00.899405 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44727
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:44727
--ranger_config_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:01.235806 22971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:01.236114 22971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:01.236241 22971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:01.243366 22971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:01.243476 22971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:01.243541 22971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:01.243618 22971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:01.243670 22971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:01.253793 22971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.110.20:50985/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:44727
--ranger_config_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:01.255110 22971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:01.256465 22971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:01.267917 22976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:01.268034 22979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:01.268067 22977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:01.268837 22971 server_base.cc:1034] running on GCE node
I20260926 16:40:01.269590 22971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:01.271054 22971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:01.272274 22971 hybrid_clock.cc:648] HybridClock initialized: now 1790440801272221 us; error 33 us; skew 500 ppm
I20260926 16:40:01.272661 22971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:01 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440801, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:01.275566 22971 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:01.277045 22971 webserver.cc:460] Webserver started at http://127.10.110.62:43853/ using document root <none> and password file <none>
I20260926 16:40:01.277652 22971 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:01.277747 22971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:01.278004 22971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:40:01 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440801, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:01.339344 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "7897a8639f554ff9bc50b8ba8384e5c7"
format_stamp: "Formatted at 2026-09-26 16:40:01 on dist-test-slave-whjp"
server_key: "709d456a61471bfb11880297faeae9e8"
server_key_iv: "68645d4637e41547e56db5bf9443b267"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:01.340265 22971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "7897a8639f554ff9bc50b8ba8384e5c7"
format_stamp: "Formatted at 2026-09-26 16:40:01 on dist-test-slave-whjp"
server_key: "709d456a61471bfb11880297faeae9e8"
server_key_iv: "68645d4637e41547e56db5bf9443b267"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:01.345656 22971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:40:01.365569 22986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:01.367027 22971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 16:40:01.367187 22971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "7897a8639f554ff9bc50b8ba8384e5c7"
format_stamp: "Formatted at 2026-09-26 16:40:01 on dist-test-slave-whjp"
server_key: "709d456a61471bfb11880297faeae9e8"
server_key_iv: "68645d4637e41547e56db5bf9443b267"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:01.367395 22971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:01.382004 22971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:01.385854 22971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:01.386137 22971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:01.403652 22971 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:44727
I20260926 16:40:01.403656 23037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:44727 every 8 connection(s)
I20260926 16:40:01.404983 22971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:01.407799 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 22971
I20260926 16:40:01.408035 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:40:01.408774 23038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:01.408852 10680 external_mini_cluster.cc:1428] Setting key 5ab76f404b6d31d13ba228bdd0c0c3c2
I20260926 16:40:01.422912 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Bootstrap starting.
I20260926 16:40:01.426225 23038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:01.427197 23038 log.cc:824] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:01.429816 23038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: No bootstrap required, opened a new log
Sep 26 16:40:01 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:01.437528 23038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:01.437920 23038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:01.438022 23038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897a8639f554ff9bc50b8ba8384e5c7, State: Initialized, Role: FOLLOWER
I20260926 16:40:01.438593 23038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:01.438812 23038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:01.438935 23038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:01.439081 23038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:01.441213 23038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:01.441706 23038 leader_election.cc:302] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7897a8639f554ff9bc50b8ba8384e5c7; no voters:
I20260926 16:40:01.442308 23038 leader_election.cc:288] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:40:01.442428 23043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:01.443010 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:01.410961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.10.110.62:44727)
0926 16:40:01.420443 (+ 9482us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:01.420460 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:01.420817 (+ 357us) server_negotiation.cc:407] Connection header received
0926 16:40:01.422596 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:01.422620 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:01.423033 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:01.423424 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:01.424795 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:01.426492 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:01.427770 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:01.428500 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:01.428655 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:01.435960 (+ 7305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:01.435968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:01.435984 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:01.436028 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:01.438972 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:01.439645 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:01.439654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:01.439672 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:01.439817 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:01.440191 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:01.440196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:01.440199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:01.440732 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:01.440986 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:01.441426 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 16:40:01.441810 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9206,"thread_start_us":8975,"threads_started":1}
I20260926 16:40:01.443442 23043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 LEADER]: Becoming Leader. State: Replica: 7897a8639f554ff9bc50b8ba8384e5c7, State: Running, Role: LEADER
I20260926 16:40:01.444077 23043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:01.444509 23038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:01.447540 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } } }
I20260926 16:40:01.447925 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7897a8639f554ff9bc50b8ba8384e5c7. Latest consensus state: current_term: 1 leader_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } } }
I20260926 16:40:01.448484 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:01.448789 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:01.455749 23049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:01.460812 23049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:01.461397 23038 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:40:01.478032 23049 catalog_manager.cc:1347] Generated new cluster ID: 6bbd94a455804419b0236214a9162c67
I20260926 16:40:01.478365 23049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:01.497586 23049 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:40:01.499048 23049 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:01.518247 23049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Generated new TSK 0
I20260926 16:40:01.519156 23049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:40:02.289077 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44727
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:40:02 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440802, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:40:02.695235 23083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:02.695632 23083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:02.695806 23083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:02.708660 23083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:02.708806 23083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:02.709044 23083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:40:02.724480 23083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.110.20:50985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:02.726255 23083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:02.728005 23083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:02.744767 23094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:02.745651 23093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:02.746168 23096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:02.747262 23083 server_base.cc:1034] running on GCE node
I20260926 16:40:02.747906 23083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:02.748871 23083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:02.750138 23083 hybrid_clock.cc:648] HybridClock initialized: now 1790440802750081 us; error 57 us; skew 500 ppm
I20260926 16:40:02.750449 23083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:02 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440802, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:02.756528 23083 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:40:02.765808 23083 webserver.cc:460] Webserver started at http://127.10.110.1:40159/ using document root <none> and password file <none>
I20260926 16:40:02.766897 23083 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:02.767087 23083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:02.767534 23083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:40:02 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440802, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:02.789160 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "86045369158249d6b9b43ab341a599d2"
format_stamp: "Formatted at 2026-09-26 16:40:02 on dist-test-slave-whjp"
server_key: "fc7ef92966a74f3a834455a16d7b0ca9"
server_key_iv: "c21b695dec8343579ddc9a2e45843bab"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:02.792145 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "86045369158249d6b9b43ab341a599d2"
format_stamp: "Formatted at 2026-09-26 16:40:02 on dist-test-slave-whjp"
server_key: "fc7ef92966a74f3a834455a16d7b0ca9"
server_key_iv: "c21b695dec8343579ddc9a2e45843bab"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:02.806185 23083 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260926 16:40:02.835531 23103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:02.837599 23083 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.006s
I20260926 16:40:02.837882 23083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "86045369158249d6b9b43ab341a599d2"
format_stamp: "Formatted at 2026-09-26 16:40:02 on dist-test-slave-whjp"
server_key: "fc7ef92966a74f3a834455a16d7b0ca9"
server_key_iv: "c21b695dec8343579ddc9a2e45843bab"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:02.838241 23083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:02.862227 23083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:02.870306 23083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:02.870589 23083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:02.871611 23083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:02.873159 23083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:02.873250 23083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:02.873373 23083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:02.873420 23083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:02.940825 23083 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:42567
I20260926 16:40:02.940865 23215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:42567 every 8 connection(s)
I20260926 16:40:02.942492 23083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:40:02.942993 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23083
I20260926 16:40:02.943192 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:40:02.944092 10680 external_mini_cluster.cc:1428] Setting key d654d3034c8d6510a96e7f8b47512683
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Sep 26 16:40:02 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440802, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:40:02.985093 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:02.952154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:54489 (local address 127.10.110.62:44727)
0926 16:40:02.956011 (+ 3857us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:02.956019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:02.961811 (+ 5792us) server_negotiation.cc:407] Connection header received
0926 16:40:02.964591 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:02.964598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:02.964723 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:02.964852 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:02.967035 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:02.968251 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:02.970813 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:02.971526 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:02.971635 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:02.977382 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:02.977390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:02.977396 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:02.977441 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:02.981037 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:02.982105 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:02.982112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:02.982127 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:02.982289 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:02.982852 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:02.982857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:02.982860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:02.983245 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:02.983368 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:02.984461 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0926 16:40:02.984697 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3719,"thread_start_us":3581,"threads_started":1}
I20260926 16:40:02.987319 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:02.952683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44727 (local address 127.10.110.1:54489)
0926 16:40:02.961608 (+ 8925us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:02.961666 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:02.964229 (+ 2563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:02.965185 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:02.965199 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:02.965934 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:02.966806 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:02.966834 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:02.968482 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:02.968487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:02.970638 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:02.970654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:02.972081 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:02.972089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:02.972387 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:02.973249 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:02.973281 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:02.977064 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:02.981257 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:02.981268 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:02.981291 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:02.981948 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:02.982447 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:02.982452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:02.982456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:02.982718 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:02.983480 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:02.983489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:02.983983 (+ 494us) client_negotiation.cc:769] Sending connection context
0926 16:40:02.985342 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0926 16:40:02.985907 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8593,"thread_start_us":8387,"threads_started":1}
I20260926 16:40:02.989431 23216 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44727
I20260926 16:40:02.989775 23216 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:02.990675 23216 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:02.993252 23003 ts_manager.cc:194] Registered new tserver with Master: 86045369158249d6b9b43ab341a599d2 (127.10.110.1:42567)
I20260926 16:40:02.995282 23003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:54489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:40:03.014724 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44727
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:40:03 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440802, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
Sep 26 16:40:03 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440802, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
W20260926 16:40:03.505376 23222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:03.505687 23222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:03.505852 23222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:03.514679 23222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:03.514878 23222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:03.515115 23222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:40:03.528465 23222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.110.20:50985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:03.530301 23222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:03.531977 23222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:03.565150 23229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:03.566318 23230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:03.568181 23232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:03.568841 23222 server_base.cc:1034] running on GCE node
I20260926 16:40:03.569603 23222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:03.577850 23222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:03.579169 23222 hybrid_clock.cc:648] HybridClock initialized: now 1790440803579073 us; error 97 us; skew 500 ppm
I20260926 16:40:03.579617 23222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:03 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440803, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:03.587248 23222 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:40:03.609766 23222 webserver.cc:460] Webserver started at http://127.10.110.2:42619/ using document root <none> and password file <none>
I20260926 16:40:03.610952 23222 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:03.611187 23222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:03.611652 23222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:40:03 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440803, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:03.638911 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "e23893e5125442d994690861b0ea4ae2"
format_stamp: "Formatted at 2026-09-26 16:40:03 on dist-test-slave-whjp"
server_key: "8cadcdb37aa8bded3a0c89d4d07820c0"
server_key_iv: "f367d89f63774a7596aef135f48c038c"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:03.640321 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "e23893e5125442d994690861b0ea4ae2"
format_stamp: "Formatted at 2026-09-26 16:40:03 on dist-test-slave-whjp"
server_key: "8cadcdb37aa8bded3a0c89d4d07820c0"
server_key_iv: "f367d89f63774a7596aef135f48c038c"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:03.658136 23222 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.012s
I20260926 16:40:03.710952 23239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:03.713164 23222 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.005s sys 0.000s
I20260926 16:40:03.713472 23222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "e23893e5125442d994690861b0ea4ae2"
format_stamp: "Formatted at 2026-09-26 16:40:03 on dist-test-slave-whjp"
server_key: "8cadcdb37aa8bded3a0c89d4d07820c0"
server_key_iv: "f367d89f63774a7596aef135f48c038c"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:03.713840 23222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:03.784097 23222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:03.789705 23222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:03.790113 23222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:03.803377 23222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:03.811802 23222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:03.812033 23222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:40:03.812216 23222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:03.812317 23222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:03.938202 23222 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:39389
I20260926 16:40:03.938251 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:39389 every 8 connection(s)
I20260926 16:40:03.939759 23222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:40:03.944008 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23222
I20260926 16:40:03.944175 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:40:03.944737 10680 external_mini_cluster.cc:1428] Setting key a687e799508297c71026a3fefa520aea
I20260926 16:40:03.999102 23216 heartbeater.cc:503] Master 127.10.110.62:44727 was elected leader, sending a full tablet report...
Sep 26 16:40:03 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440803, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:40:04.007949 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:03.952086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:51293 (local address 127.10.110.62:44727)
0926 16:40:03.979017 (+ 26931us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:03.979024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:03.980258 (+ 1234us) server_negotiation.cc:407] Connection header received
0926 16:40:03.983807 (+ 3549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:03.983813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:03.983940 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:03.984033 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:03.992024 (+ 7991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:03.993172 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:03.995716 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:03.996330 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:03.996470 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:04.000784 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:04.000790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:04.000794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:04.000829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:04.004251 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.005186 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.005192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.005195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.005335 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.005836 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.005841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.005844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.006171 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:04.006287 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:04.007307 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0926 16:40:04.007544 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26774,"thread_start_us":26956,"threads_started":1}
I20260926 16:40:04.010114 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:03.951474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44727 (local address 127.10.110.2:51293)
0926 16:40:03.979757 (+ 28283us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:03.979849 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:03.982470 (+ 2621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:03.984632 (+ 2162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:03.984645 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:03.985317 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:03.986142 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:03.986168 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:03.993399 (+ 7231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:03.993405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:03.995539 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:03.995560 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:03.996592 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:03.996599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:03.996902 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:03.997704 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:03.997734 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:04.000530 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:04.004434 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:04.004443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:04.004464 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:04.005053 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:04.005469 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:04.005474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:04.005477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:04.005721 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:04.006378 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:04.006386 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:04.006858 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 16:40:04.008200 (+ 1342us) client_negotiation.cc:240] Negotiation successful
0926 16:40:04.008764 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27908,"thread_start_us":31239,"threads_started":1}
I20260926 16:40:04.011817 23359 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:40:04.012149 23359 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:04.012959 23359 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:04.016765 23003 ts_manager.cc:194] Registered new tserver with Master: e23893e5125442d994690861b0ea4ae2 (127.10.110.2:39389)
I20260926 16:40:04.017870 23003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:51293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:40:04.047219 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44727
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:04.391402 23365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:04.391670 23365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:04.391800 23365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:04.398926 23365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:04.399031 23365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:04.399179 23365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:40:04.408993 23365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.110.20:50985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:04.410328 23365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:04.411598 23365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:04.423933 23373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:04.423923 23370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:04.424453 23371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:04.425289 23365 server_base.cc:1034] running on GCE node
I20260926 16:40:04.425887 23365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:04.426802 23365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:04.427978 23365 hybrid_clock.cc:648] HybridClock initialized: now 1790440804427937 us; error 39 us; skew 500 ppm
I20260926 16:40:04.428268 23365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:04 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440804, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:04.431080 23365 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:40:04.432502 23365 webserver.cc:460] Webserver started at http://127.10.110.3:32795/ using document root <none> and password file <none>
I20260926 16:40:04.433166 23365 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:04.433259 23365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:04.433491 23365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:40:04 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440804, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:04.449767 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "4924888ca3e8456d8b8bf0e8e0948995"
format_stamp: "Formatted at 2026-09-26 16:40:04 on dist-test-slave-whjp"
server_key: "137d583336695e19a3071bb9c3cb6f7f"
server_key_iv: "914bad48fc31f50b1f1fc8a069c07123"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:04.450724 23365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "4924888ca3e8456d8b8bf0e8e0948995"
format_stamp: "Formatted at 2026-09-26 16:40:04 on dist-test-slave-whjp"
server_key: "137d583336695e19a3071bb9c3cb6f7f"
server_key_iv: "914bad48fc31f50b1f1fc8a069c07123"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:04.456286 23365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 16:40:04.473367 23380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:04.474804 23365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:04.474972 23365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "4924888ca3e8456d8b8bf0e8e0948995"
format_stamp: "Formatted at 2026-09-26 16:40:04 on dist-test-slave-whjp"
server_key: "137d583336695e19a3071bb9c3cb6f7f"
server_key_iv: "914bad48fc31f50b1f1fc8a069c07123"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:04.475186 23365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:04.507625 23365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:04.511507 23365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:04.511788 23365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:04.512692 23365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:04.513963 23365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:04.514036 23365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:04.514153 23365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:04.514206 23365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:04.543475 23365 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:33865
I20260926 16:40:04.543509 23492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:33865 every 8 connection(s)
I20260926 16:40:04.544860 23365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:40:04.545286 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23365
I20260926 16:40:04.545433 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:40:04.545946 10680 external_mini_cluster.cc:1428] Setting key 395772191c437433892d3193e9e14555
Sep 26 16:40:04 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440804, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:04.563715 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:04.547486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:47753 (local address 127.10.110.62:44727)
0926 16:40:04.547939 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:04.547945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:04.549038 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 16:40:04.550802 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:04.550806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:04.550879 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:04.550964 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:04.552335 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.553010 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.554444 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.554843 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.554926 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:04.557995 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:04.557999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:04.558003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:04.558034 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:04.560870 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.561536 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.561540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.561542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.561669 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.562069 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.562073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.562075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.562343 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:04.562472 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:04.563116 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 16:40:04.563349 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":1408,"thread_start_us":317,"threads_started":1}
I20260926 16:40:04.564615 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:04.548138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44727 (local address 127.10.110.3:47753)
0926 16:40:04.548905 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:04.548947 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:04.550574 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:04.551147 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:04.551154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:04.551560 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:04.552190 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:04.552210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.553127 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.553130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:04.554340 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:04.554353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.555007 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.555011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:04.555213 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:04.555818 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:04.555879 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:04.557784 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:04.560993 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:04.561001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:04.561015 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:04.561419 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:04.561766 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:04.561770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:04.561772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:04.561958 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:04.562460 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:04.562468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:04.562774 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 16:40:04.563001 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 16:40:04.563508 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260926 16:40:04.565816 23493 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44727
I20260926 16:40:04.566071 23493 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:04.566689 23493 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:04.567960 23003 ts_manager.cc:194] Registered new tserver with Master: 4924888ca3e8456d8b8bf0e8e0948995 (127.10.110.3:33865)
I20260926 16:40:04.568732 23003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:47753
I20260926 16:40:04.573335 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:40:04.574934 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23083
Sep 26 16:40:04 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:40:04.589883 23498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:04.575694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.10.110.1:42567)
0926 16:40:04.576241 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:04.576248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:04.576274 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:40:04.576373 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:04.576380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:04.576577 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:04.576708 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:04.578016 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.579068 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.580539 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.581089 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.581186 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:04.585033 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:04.585045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:04.585060 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:04.585096 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:04.587516 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.588185 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.588191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.588194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.588324 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.588595 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.588597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.588599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.588915 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:04.589131 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:04.589343 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 16:40:04.589532 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":261,"threads_started":1}
I20260926 16:40:04.591104 23211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:04.673175 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23083
I20260926 16:40:04.686018 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23222
Sep 26 16:40:04 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:40:04.700882 23500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:04.686903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.10.110.2:39389)
0926 16:40:04.687410 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:04.687417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:04.687442 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:40:04.687499 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:04.687506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:04.687712 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:04.687854 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:04.688963 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.689947 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.691190 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.691712 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.691813 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:04.695800 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:04.695813 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:04.695926 (+ 113us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:04.695969 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:04.698452 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.699084 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.699090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.699092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.699235 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.699508 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.699511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.699513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.699856 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:04.700080 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:04.700336 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 16:40:04.700539 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260926 16:40:04.702067 23354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:04.783646 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23222
I20260926 16:40:04.796096 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23365
Sep 26 16:40:04 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:40:04.812878 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:04.797175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.10.110.3:33865)
0926 16:40:04.797850 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:04.797858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:04.797888 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:40:04.797947 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:04.797953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:04.798166 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:04.798307 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:04.799668 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.800745 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.802074 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:04.802719 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:04.802852 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:04.807312 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:04.807332 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:04.807349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:04.807389 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:04.810035 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.810796 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.810804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.810807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.810944 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:04.811311 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:04.811314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:04.811317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:04.811660 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:04.811991 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:04.812292 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:40:04.812545 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":357,"threads_started":1}
I20260926 16:40:04.814136 23488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:04.895931 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23365
I20260926 16:40:04.907919 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 22971
I20260926 16:40:04.909224 23033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:05.027253 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 22971
I20260926 16:40:05.045047 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44727
--webserver_interface=127.10.110.62
--webserver_port=43853
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:44727
--ranger_config_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:05.370854 23505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:05.371168 23505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:05.371301 23505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:05.378788 23505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:05.378917 23505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:05.378973 23505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:05.379050 23505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:05.379113 23505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:05.389992 23505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:44727
--ranger_config_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=43853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:05.391335 23505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:05.392724 23505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:05.404191 23513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:05.404250 23510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:05.404229 23511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:05.404904 23505 server_base.cc:1034] running on GCE node
I20260926 16:40:05.405691 23505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:05.407035 23505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:05.408248 23505 hybrid_clock.cc:648] HybridClock initialized: now 1790440805408196 us; error 39 us; skew 500 ppm
I20260926 16:40:05.408602 23505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:05 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440805, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:05.411541 23505 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:05.413120 23505 webserver.cc:460] Webserver started at http://127.10.110.62:43853/ using document root <none> and password file <none>
I20260926 16:40:05.413781 23505 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:05.413899 23505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:40:05 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440805, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:05.476815 23505 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:40:05.479988 23521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:05.481451 23505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:05.481634 23505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "7897a8639f554ff9bc50b8ba8384e5c7"
format_stamp: "Formatted at 2026-09-26 16:40:01 on dist-test-slave-whjp"
server_key: "709d456a61471bfb11880297faeae9e8"
server_key_iv: "68645d4637e41547e56db5bf9443b267"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:05.482617 23505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:05.503714 23505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:05.507741 23505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:05.508057 23505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:05.525605 23505 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:44727
I20260926 16:40:05.525628 23572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:44727 every 8 connection(s)
I20260926 16:40:05.527084 23505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:05.527621 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23505
I20260926 16:40:05.528625 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:42567
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:44727
--webserver_port=40159
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:05.534142 23573 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:40:05.537539 23573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Bootstrap starting.
I20260926 16:40:05.556208 23573 log.cc:824] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:05.586056 23573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:40:05.586656 23573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Bootstrap complete.
I20260926 16:40:05.596342 23573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:05.597295 23573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7897a8639f554ff9bc50b8ba8384e5c7, State: Initialized, Role: FOLLOWER
I20260926 16:40:05.597919 23573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:05.598197 23573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:05.598335 23573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:05.598544 23573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:40:05.602576 23573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:05.603101 23573 leader_election.cc:302] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7897a8639f554ff9bc50b8ba8384e5c7; no voters:
I20260926 16:40:05.603736 23573 leader_election.cc:288] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:40:05.603910 23577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:40:05.604970 23577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [term 2 LEADER]: Becoming Leader. State: Replica: 7897a8639f554ff9bc50b8ba8384e5c7, State: Running, Role: LEADER
I20260926 16:40:05.605581 23577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } }
I20260926 16:40:05.605916 23573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:05.608084 23578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } } }
I20260926 16:40:05.608439 23578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:05.608845 23579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7897a8639f554ff9bc50b8ba8384e5c7. Latest consensus state: current_term: 2 leader_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7897a8639f554ff9bc50b8ba8384e5c7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44727 } } }
I20260926 16:40:05.609345 23579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:05.613943 23582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:05.620323 23582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:05.621968 23582 catalog_manager.cc:1259] Loaded cluster ID: 6bbd94a455804419b0236214a9162c67
I20260926 16:40:05.622174 23582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:05.626629 23582 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:05.628605 23582 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Loaded TSK: 0
I20260926 16:40:05.630874 23582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:40:05.642076 23573 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 16:40:05.983264 23575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:05.983776 23575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:05.984014 23575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:05.996412 23575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:05.996652 23575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:05.996888 23575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:40:06.012591 23575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:42567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=40159
--tserver_master_addrs=127.10.110.62:44727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:06.014437 23575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:06.016317 23575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:06.050480 23610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:06.052220 23611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:06.057142 23575 server_base.cc:1034] running on GCE node
W20260926 16:40:06.057389 23613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:06.058311 23575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:06.065980 23575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:06.067293 23575 hybrid_clock.cc:648] HybridClock initialized: now 1790440806067217 us; error 75 us; skew 500 ppm
I20260926 16:40:06.067736 23575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:06.072346 23575 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:40:06.103798 23575 webserver.cc:460] Webserver started at http://127.10.110.1:40159/ using document root <none> and password file <none>
I20260926 16:40:06.104911 23575 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:06.105094 23575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:06.138127 23575 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:40:06.152560 23621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:06.154881 23575 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.000s
I20260926 16:40:06.155115 23575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "86045369158249d6b9b43ab341a599d2"
format_stamp: "Formatted at 2026-09-26 16:40:02 on dist-test-slave-whjp"
server_key: "fc7ef92966a74f3a834455a16d7b0ca9"
server_key_iv: "c21b695dec8343579ddc9a2e45843bab"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:06.156684 23575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:06.188153 23575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:06.199100 23575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:06.199414 23575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:06.207608 23575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:06.210068 23575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:06.210186 23575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:06.210326 23575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:06.210404 23575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:06.254289 23575 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:42567
I20260926 16:40:06.254323 23734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:42567 every 8 connection(s)
I20260926 16:40:06.255776 23575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:40:06.263900 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23575
I20260926 16:40:06.264765 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:39389
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:44727
--webserver_port=42619
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:06.309583 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:06.259366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44727 (local address 127.10.110.1:58281)
0926 16:40:06.277779 (+ 18413us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:06.277854 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:06.280758 (+ 2904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:06.284280 (+ 3522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:06.284293 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:06.284992 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:06.285835 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:06.285864 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.288425 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.288430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:06.290520 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:06.290538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.292542 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.292549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:06.292833 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:06.295789 (+ 2956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:06.295859 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:06.298866 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:06.303724 (+ 4858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:06.303736 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:06.303756 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:06.304462 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:06.305093 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:06.305098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:06.305102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:06.305382 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:06.306579 (+ 1197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:06.306587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:06.307096 (+ 509us) client_negotiation.cc:769] Sending connection context
0926 16:40:06.307420 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 16:40:06.308070 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17999,"thread_start_us":18678,"threads_started":1}
I20260926 16:40:06.312618 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:06.260615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:58281 (local address 127.10.110.62:44727)
0926 16:40:06.276988 (+ 16373us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:06.277015 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:06.281249 (+ 4234us) server_negotiation.cc:407] Connection header received
0926 16:40:06.281820 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:06.281844 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:06.282433 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:06.282907 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:06.286092 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.288200 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.290730 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.291907 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.292118 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:06.299281 (+ 7163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:06.299295 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:06.299317 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:06.299376 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:06.303465 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:06.304679 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:06.304690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:06.304699 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:06.304941 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:06.305546 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:06.305552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:06.305555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:06.306303 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:06.306488 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:06.310626 (+ 4138us) server_negotiation.cc:299] Negotiation successful
0926 16:40:06.311158 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2494,"thread_start_us":2218,"threads_started":1}
I20260926 16:40:06.316208 23735 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44727
I20260926 16:40:06.316566 23735 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:06.317338 23735 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
W20260926 16:40:06.320513 23735 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:44727 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:40:06.320737 23735 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:06.321031 23735 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:06.322393 23735 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:06.322678 23735 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
W20260926 16:40:06.323896 23735 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:40:06.473248 23746 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 16:40:06.698202 23739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:06.698496 23739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:06.698647 23739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:06.708590 23739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:06.708866 23739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:06.709233 23739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:40:06.725486 23739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:39389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=42619
--tserver_master_addrs=127.10.110.62:44727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:06.727353 23739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:06.729053 23739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:06.761493 23755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:06.763034 23756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:06.770556 23758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:06.771464 23739 server_base.cc:1034] running on GCE node
I20260926 16:40:06.772334 23739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:06.773756 23739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:06.775027 23739 hybrid_clock.cc:648] HybridClock initialized: now 1790440806774963 us; error 54 us; skew 500 ppm
I20260926 16:40:06.775429 23739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:06.779588 23739 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:40:06.781675 23739 webserver.cc:460] Webserver started at http://127.10.110.2:42619/ using document root <none> and password file <none>
I20260926 16:40:06.782646 23739 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:06.782811 23739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:06.817721 23739 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.004s sys 0.008s
I20260926 16:40:06.822167 23766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:06.824358 23739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 16:40:06.824574 23739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "e23893e5125442d994690861b0ea4ae2"
format_stamp: "Formatted at 2026-09-26 16:40:03 on dist-test-slave-whjp"
server_key: "8cadcdb37aa8bded3a0c89d4d07820c0"
server_key_iv: "f367d89f63774a7596aef135f48c038c"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:06.826035 23739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:06.854329 23739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:06.859973 23739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:06.860481 23739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:06.863024 23739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:06.864856 23739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:06.864966 23739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:06.865089 23739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:06.865154 23739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:06.906316 23739 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:39389
I20260926 16:40:06.906397 23879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:39389 every 8 connection(s)
I20260926 16:40:06.908322 23739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:40:06.918642 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23739
I20260926 16:40:06.919430 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.110.20:50985/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:33865
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:44727
--webserver_port=32795
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:40:06 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:06.970311 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:06.911819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44727 (local address 127.10.110.2:47817)
0926 16:40:06.943552 (+ 31733us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:06.943623 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:06.947943 (+ 4320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:06.949229 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:06.949243 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:06.949954 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:06.950923 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:06.950955 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.953012 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.953018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:06.955031 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:06.955050 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.956152 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.956161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:06.956495 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:06.957355 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:06.957388 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:06.960446 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:06.964797 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:06.964808 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:06.964834 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:06.965569 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:06.966105 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:06.966110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:06.966114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:06.966443 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:06.967259 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:06.967268 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:06.967778 (+ 510us) client_negotiation.cc:769] Sending connection context
0926 16:40:06.968159 (+ 381us) client_negotiation.cc:240] Negotiation successful
0926 16:40:06.968776 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30791,"thread_start_us":31321,"threads_started":1}
I20260926 16:40:06.971503 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:06.912148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:47817 (local address 127.10.110.62:44727)
0926 16:40:06.945158 (+ 33010us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:06.945194 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:06.945223 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:40:06.948490 (+ 3267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:06.948495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:06.948589 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:06.948720 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:06.951657 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.952783 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.955248 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:06.955911 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:06.956045 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:06.960784 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:06.960792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:06.960797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:06.960844 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:06.964559 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:06.965757 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:06.965763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:06.965767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:06.965941 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:06.966608 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:06.966613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:06.966617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:06.967005 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:06.967148 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:06.970841 (+ 3693us) server_negotiation.cc:299] Negotiation successful
0926 16:40:06.971119 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32866,"thread_start_us":34037,"threads_started":1}
I20260926 16:40:06.974422 23880 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44727
I20260926 16:40:06.974819 23880 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:06.975924 23880 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:06.978245 23538 ts_manager.cc:194] Registered new tserver with Master: e23893e5125442d994690861b0ea4ae2 (127.10.110.2:39389)
I20260926 16:40:06.980551 23538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:47817
Sep 26 16:40:07 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:07.323992 23735 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:07.324425 23735 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:07.325654 23538 ts_manager.cc:194] Registered new tserver with Master: 86045369158249d6b9b43ab341a599d2 (127.10.110.1:42567)
I20260926 16:40:07.326812 23538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:58281
W20260926 16:40:07.402029 23884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:07.402354 23884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:07.402498 23884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:07.414358 23884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:07.414530 23884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:07.414721 23884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:40:07.425830 23884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.110.20:50985/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:33865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=32795
--tserver_master_addrs=127.10.110.62:44727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:07.427239 23884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:07.428580 23884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:07.445188 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:07.446236 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:07.461611 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:07.461875 23884 server_base.cc:1034] running on GCE node
I20260926 16:40:07.462697 23884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:07.463912 23884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:07.465165 23884 hybrid_clock.cc:648] HybridClock initialized: now 1790440807465097 us; error 60 us; skew 500 ppm
I20260926 16:40:07.465577 23884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:07 dist-test-slave-whjp krb5kdc[22527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440807, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:07.468943 23884 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:40:07.473759 23884 webserver.cc:460] Webserver started at http://127.10.110.3:32795/ using document root <none> and password file <none>
I20260926 16:40:07.474546 23884 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:07.474685 23884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:40:07 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440807, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:07.498521 23884 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 16:40:07.501785 23900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:07.503306 23884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:40:07.503490 23884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "4924888ca3e8456d8b8bf0e8e0948995"
format_stamp: "Formatted at 2026-09-26 16:40:04 on dist-test-slave-whjp"
server_key: "137d583336695e19a3071bb9c3cb6f7f"
server_key_iv: "914bad48fc31f50b1f1fc8a069c07123"
server_key_version: "kuduclusterkey@0"
I20260926 16:40:07.504612 23884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:07.520198 23884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:07.524340 23884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:07.524645 23884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:07.538138 23884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:07.540375 23884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:07.540575 23884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:40:07.540746 23884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:07.540874 23884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:40:07 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440806, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for HTTP/127.10.110.20@KRBTEST.COM
I20260926 16:40:07.640797 23884 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:33865
I20260926 16:40:07.640841 24013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:33865 every 8 connection(s)
I20260926 16:40:07.642493 23884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:40:07.643460 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 23884
Sep 26 16:40:07 dist-test-slave-whjp krb5kdc[22527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440807, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:07.704668 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.648622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.10.110.62:44727)
0926 16:40:07.678570 (+ 29948us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:07.678578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:07.678606 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:40:07.678692 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:07.678698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:07.678816 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:07.678953 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:07.683166 (+ 4213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.684496 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.686619 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.687269 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.687454 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.691276 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:07.691283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:07.691288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:07.691337 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:07.699870 (+ 8533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:07.701178 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:07.701184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:07.701187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:07.701356 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:07.701915 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:07.701920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:07.701923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:07.702456 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:07.703733 (+ 1277us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:07.703906 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 16:40:07.704249 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29772,"thread_start_us":37085,"threads_started":1}
I20260926 16:40:07.711036 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.653048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:51165 (local address 127.10.110.62:44727)
0926 16:40:07.686040 (+ 32992us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:07.686046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:07.686065 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:40:07.686131 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:07.686137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:07.686240 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:07.686398 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:07.689848 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.691056 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.695283 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.695973 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.696099 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.703648 (+ 7549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:07.703655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:07.703659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:07.703696 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:07.707154 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:07.708141 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:07.708147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:07.708151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:07.708340 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:07.708809 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:07.708814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:07.708817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:07.709119 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:07.709247 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:07.710410 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0926 16:40:07.710666 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32858,"thread_start_us":26057,"threads_started":1}
I20260926 16:40:07.711872 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.652454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44727 (local address 127.10.110.3:51165)
0926 16:40:07.679865 (+ 27411us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:07.679927 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:07.682563 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:07.688036 (+ 5473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:07.688051 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:07.688733 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:07.689590 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.689620 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.692168 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.692175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.694244 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.694262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.696490 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.696499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.699311 (+ 2812us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.700462 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:07.700503 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:07.703338 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:07.707326 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:07.707336 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:07.707356 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:07.708017 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:07.708451 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:07.708456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:07.708460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:07.708676 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:07.709377 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:07.709386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:07.709834 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 16:40:07.710171 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 16:40:07.710637 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27010,"thread_start_us":32134,"threads_started":1}
I20260926 16:40:07.713351 24014 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44727
I20260926 16:40:07.713645 24014 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:07.714422 24014 heartbeater.cc:511] Master 127.10.110.62:44727 requested a full tablet report, sending...
I20260926 16:40:07.715956 23538 ts_manager.cc:194] Registered new tserver with Master: 4924888ca3e8456d8b8bf0e8e0948995 (127.10.110.3:33865)
I20260926 16:40:07.716881 23538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:51165
I20260926 16:40:07.721827 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:40:07.796015 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.778056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.10.110.62:44727)
0926 16:40:07.778224 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:07.778230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:07.778349 (+ 119us) server_negotiation.cc:407] Connection header received
0926 16:40:07.778656 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:07.778663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:07.778749 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:07.778820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:07.781077 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.782230 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.784337 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.785044 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.785164 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.789289 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:07.789297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:07.789302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:07.789349 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:07.792851 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:07.793852 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:07.793858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:07.793862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:07.794008 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:07.794393 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:07.794398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:07.794402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:07.794696 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:07.794912 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:07.795247 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 16:40:07.795461 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 16:40:07.802322 23538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:40:07.812755 23538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:40:07.902283 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.863472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:33865 (local address 127.0.0.1:40390)
0926 16:40:07.889812 (+ 26340us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:07.889870 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:07.890101 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:07.892928 (+ 2827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:07.892935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:07.892963 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:07.893366 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.893383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.896779 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.896785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.898831 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.898841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.901121 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.901127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.901405 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.901464 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 16:40:07.901612 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 16:40:07.901826 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26002,"mutex_wait_us":30,"thread_start_us":28881,"threads_started":1}
I20260926 16:40:07.902804 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.863754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.10.110.2:39389)
0926 16:40:07.887916 (+ 24162us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:07.887927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:07.892789 (+ 4862us) server_negotiation.cc:407] Connection header received
0926 16:40:07.892906 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:07.892914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:07.893180 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:07.893383 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:07.894474 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.896120 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.899129 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.900556 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.900683 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.900893 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:07.902306 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0926 16:40:07.902496 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23934,"thread_start_us":24103,"threads_started":1}
I20260926 16:40:07.905467 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.864144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.10.110.3:33865)
0926 16:40:07.889129 (+ 24985us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:07.889140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:07.891131 (+ 1991us) server_negotiation.cc:407] Connection header received
0926 16:40:07.891310 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:07.891319 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:07.891626 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:07.891868 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:07.893633 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.896623 (+ 2990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.899009 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.900556 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.900683 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.900928 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:07.902685 (+ 1757us) server_negotiation.cc:299] Negotiation successful
0926 16:40:07.905094 (+ 2409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24812,"thread_start_us":24659,"threads_started":1}
I20260926 16:40:07.903779 24030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.865356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:39389 (local address 127.0.0.1:60954)
0926 16:40:07.892625 (+ 27269us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:07.892644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:07.892778 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:07.893941 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:07.893945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:07.893971 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:07.894288 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.894302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.896305 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.896311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.898765 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.898782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.901163 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.901168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.901693 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.901721 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 16:40:07.901789 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 16:40:07.901894 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27136,"mutex_wait_us":323,"thread_start_us":25006,"threads_started":1}
I20260926 16:40:07.913717 23814 tablet_service.cc:1467] Processing CreateTablet for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf (DEFAULT_TABLE table=test-table [id=55bd32022b1d4a1e8acd4c95464d0286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:07.915555 23814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:07.917651 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.893741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:42567 (local address 127.0.0.1:49774)
0926 16:40:07.904690 (+ 10949us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:07.904713 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:07.904851 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:07.907349 (+ 2498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:07.907354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:07.907384 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:07.907545 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.907555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.911544 (+ 3989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.911552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.913361 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:07.913383 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.916863 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.916869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:07.917086 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.917125 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:40:07.917198 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 16:40:07.917356 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10837,"threads_started":1}
I20260926 16:40:07.918939 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:07.893195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.10.110.1:42567)
0926 16:40:07.906343 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:07.906353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:07.906385 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:40:07.906495 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:07.906503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:07.906813 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:07.907079 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:07.907719 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.911355 (+ 3636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.913611 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:07.916661 (+ 3050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:07.918102 (+ 1441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:07.918363 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:07.918440 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 16:40:07.918618 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12974,"thread_start_us":14654,"threads_started":1}
I20260926 16:40:07.932864 23670 tablet_service.cc:1467] Processing CreateTablet for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf (DEFAULT_TABLE table=test-table [id=55bd32022b1d4a1e8acd4c95464d0286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:07.934197 23670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:07.934976 24037 tablet_bootstrap.cc:492] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Bootstrap starting.
I20260926 16:40:07.938057 23948 tablet_service.cc:1467] Processing CreateTablet for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf (DEFAULT_TABLE table=test-table [id=55bd32022b1d4a1e8acd4c95464d0286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:07.939553 23948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:07.939759 24037 tablet_bootstrap.cc:654] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:07.941425 24037 log.cc:824] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:07.953815 24037 tablet_bootstrap.cc:492] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: No bootstrap required, opened a new log
I20260926 16:40:07.954129 24039 tablet_bootstrap.cc:492] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Bootstrap starting.
I20260926 16:40:07.954361 24037 ts_tablet_manager.cc:1397] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260926 16:40:07.959216 24039 tablet_bootstrap.cc:654] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:07.960589 24039 log.cc:824] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:07.962328 24041 tablet_bootstrap.cc:492] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Bootstrap starting.
I20260926 16:40:07.966282 24041 tablet_bootstrap.cc:654] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:07.967686 24041 log.cc:824] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:07.978256 24037 raft_consensus.cc:348] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:07.979429 24037 raft_consensus.cc:374] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:07.979558 24037 raft_consensus.cc:729] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23893e5125442d994690861b0ea4ae2, State: Initialized, Role: FOLLOWER
I20260926 16:40:07.980374 24037 consensus_queue.cc:260] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:07.982834 23880 heartbeater.cc:503] Master 127.10.110.62:44727 was elected leader, sending a full tablet report...
I20260926 16:40:07.983551 24037 ts_tablet_manager.cc:1428] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Time spent starting tablet: real 0.029s user 0.014s sys 0.000s
I20260926 16:40:07.984032 24039 tablet_bootstrap.cc:492] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: No bootstrap required, opened a new log
I20260926 16:40:07.984336 24039 ts_tablet_manager.cc:1397] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.000s
I20260926 16:40:07.985778 24041 tablet_bootstrap.cc:492] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: No bootstrap required, opened a new log
I20260926 16:40:07.986050 24041 ts_tablet_manager.cc:1397] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.008s
I20260926 16:40:07.994968 24039 raft_consensus.cc:348] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:07.995461 24039 raft_consensus.cc:374] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:07.995568 24039 raft_consensus.cc:729] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86045369158249d6b9b43ab341a599d2, State: Initialized, Role: FOLLOWER
I20260926 16:40:07.996307 24039 consensus_queue.cc:260] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:07.996814 24041 raft_consensus.cc:348] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:07.997277 24041 raft_consensus.cc:374] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:07.997375 24041 raft_consensus.cc:729] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4924888ca3e8456d8b8bf0e8e0948995, State: Initialized, Role: FOLLOWER
I20260926 16:40:07.997511 23735 heartbeater.cc:503] Master 127.10.110.62:44727 was elected leader, sending a full tablet report...
I20260926 16:40:07.998199 24039 ts_tablet_manager.cc:1428] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 16:40:07.998026 24041 consensus_queue.cc:260] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:07.999174 24014 heartbeater.cc:503] Master 127.10.110.62:44727 was elected leader, sending a full tablet report...
I20260926 16:40:07.999857 24041 ts_tablet_manager.cc:1428] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260926 16:40:08.035873 23736 tablet.cc:2369] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:08.083667 24051 raft_consensus.cc:482] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:08.083976 24051 raft_consensus.cc:504] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:08.085654 24051 leader_election.cc:288] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e23893e5125442d994690861b0ea4ae2 (127.10.110.2:39389), 86045369158249d6b9b43ab341a599d2 (127.10.110.1:42567)
I20260926 16:40:08.092275 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.085960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:39389 (local address 127.10.110.3:33725)
0926 16:40:08.086165 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:08.086186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:08.086341 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:08.087391 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:08.087396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:08.087416 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:08.087617 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:08.087630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.088836 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.088842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:08.090344 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:08.090357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.091478 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.091484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:08.091662 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.091691 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:40:08.091793 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:40:08.091978 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:40:08.092295 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.086172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:33725 (local address 127.10.110.2:39389)
0926 16:40:08.086372 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:08.086378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:08.086411 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:40:08.086499 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:08.086504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:08.086575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:08.086660 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:08.087770 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.088708 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.090528 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.091307 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.091459 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.091539 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:08.091880 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 16:40:08.092034 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:40:08.094004 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.087072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:59453 (local address 127.10.110.1:42567)
0926 16:40:08.087280 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:08.087285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:08.087372 (+ 87us) server_negotiation.cc:407] Connection header received
0926 16:40:08.087577 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:08.087583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:08.087660 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:08.087760 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:08.088336 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.089359 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.091781 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.092958 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.093250 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.093331 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:08.093616 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 16:40:08.093757 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:40:08.093811 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065ca5b75c0d4f78b8c25e3ef9e56edf" candidate_uuid: "4924888ca3e8456d8b8bf0e8e0948995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23893e5125442d994690861b0ea4ae2" is_pre_election: true
I20260926 16:40:08.094033 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.086550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:42567 (local address 127.10.110.3:59453)
0926 16:40:08.087251 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:08.087270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:08.087390 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:08.087775 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:08.087778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:08.087802 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:08.088165 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:08.088176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.089538 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.089543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:08.091584 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:08.091601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.093264 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.093270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:08.093438 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.093473 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:40:08.093571 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:40:08.093712 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":304,"threads_started":1}
I20260926 16:40:08.094446 23834 raft_consensus.cc:2454] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4924888ca3e8456d8b8bf0e8e0948995 in term 0.
I20260926 16:40:08.095192 23902 leader_election.cc:302] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4924888ca3e8456d8b8bf0e8e0948995, e23893e5125442d994690861b0ea4ae2; no voters:
I20260926 16:40:08.095652 24051 raft_consensus.cc:2781] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:40:08.095791 24051 raft_consensus.cc:482] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:40:08.095930 24051 raft_consensus.cc:3037] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:08.095714 23690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065ca5b75c0d4f78b8c25e3ef9e56edf" candidate_uuid: "4924888ca3e8456d8b8bf0e8e0948995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86045369158249d6b9b43ab341a599d2" is_pre_election: true
I20260926 16:40:08.096443 23690 raft_consensus.cc:2454] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4924888ca3e8456d8b8bf0e8e0948995 in term 0.
I20260926 16:40:08.098363 24051 raft_consensus.cc:504] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:08.098980 24051 leader_election.cc:288] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [CANDIDATE]: Term 1 election: Requested vote from peers e23893e5125442d994690861b0ea4ae2 (127.10.110.2:39389), 86045369158249d6b9b43ab341a599d2 (127.10.110.1:42567)
I20260926 16:40:08.099529 23690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065ca5b75c0d4f78b8c25e3ef9e56edf" candidate_uuid: "4924888ca3e8456d8b8bf0e8e0948995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86045369158249d6b9b43ab341a599d2"
I20260926 16:40:08.099573 23834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065ca5b75c0d4f78b8c25e3ef9e56edf" candidate_uuid: "4924888ca3e8456d8b8bf0e8e0948995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23893e5125442d994690861b0ea4ae2"
I20260926 16:40:08.099731 23690 raft_consensus.cc:3037] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:08.099788 23834 raft_consensus.cc:3037] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:08.102016 23690 raft_consensus.cc:2454] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4924888ca3e8456d8b8bf0e8e0948995 in term 1.
I20260926 16:40:08.102547 23903 leader_election.cc:302] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4924888ca3e8456d8b8bf0e8e0948995, 86045369158249d6b9b43ab341a599d2; no voters:
I20260926 16:40:08.102995 24051 raft_consensus.cc:2781] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:08.103027 23834 raft_consensus.cc:2454] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4924888ca3e8456d8b8bf0e8e0948995 in term 1.
I20260926 16:40:08.104085 24051 raft_consensus.cc:686] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 1 LEADER]: Becoming Leader. State: Replica: 4924888ca3e8456d8b8bf0e8e0948995, State: Running, Role: LEADER
I20260926 16:40:08.104578 24051 consensus_queue.cc:237] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } }
I20260926 16:40:08.109791 23537 catalog_manager.cc:5495] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4924888ca3e8456d8b8bf0e8e0948995 (127.10.110.3). New cstate: current_term: 1 leader_uuid: "4924888ca3e8456d8b8bf0e8e0948995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4924888ca3e8456d8b8bf0e8e0948995" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 } health_report { overall_health: UNKNOWN } } }
I20260926 16:40:08.174399 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.168444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.10.110.3:33865)
0926 16:40:08.168643 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:08.168650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:08.168682 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:40:08.168876 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:08.168881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:08.168983 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:08.169143 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:40:08.169460 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.170529 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.171881 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.172435 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.172554 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.173023 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:40:08.173706 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:40:08.173843 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:08.173974 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 16:40:08.174062 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 16:40:08.177745 24019 tablet.cc:2369] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:08.183074 23690 raft_consensus.cc:1261] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 1 FOLLOWER]: Refusing update from remote peer 4924888ca3e8456d8b8bf0e8e0948995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:08.183161 23834 raft_consensus.cc:1261] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 1 FOLLOWER]: Refusing update from remote peer 4924888ca3e8456d8b8bf0e8e0948995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:08.184130 24053 consensus_queue.cc:1035] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e23893e5125442d994690861b0ea4ae2" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 39389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:08.184480 24051 consensus_queue.cc:1035] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86045369158249d6b9b43ab341a599d2" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:40:08.202672 23881 tablet.cc:2369] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:08.203764 24057 mvcc.cc:205] Tried to move back new op lower bound from 7333645550307430400 to 7333645550000357376. Current Snapshot: MvccSnapshot[applied={T|T < 7333645550307430400}]
I20260926 16:40:08.224835 23538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:08.225091 23538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:08.229087 23538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7897a8639f554ff9bc50b8ba8384e5c7: Sending DeleteTablet for 3 replicas of tablet 065ca5b75c0d4f78b8c25e3ef9e56edf
I20260926 16:40:08.230384 23814 tablet_service.cc:1514] Processing DeleteTablet for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:08 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:60954
I20260926 16:40:08.230427 23948 tablet_service.cc:1514] Processing DeleteTablet for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:08 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:40390
I20260926 16:40:08.231439 24065 tablet_replica.cc:330] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: stopping tablet replica
I20260926 16:40:08.231962 23670 tablet_service.cc:1514] Processing DeleteTablet for tablet 065ca5b75c0d4f78b8c25e3ef9e56edf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:08 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:49774
I20260926 16:40:08.232008 24065 raft_consensus.cc:2229] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:08.232807 24065 raft_consensus.cc:2258] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:08.235378 24065 ts_tablet_manager.cc:1905] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:08.242992 24065 ts_tablet_manager.cc:1918] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:08.243315 24065 log.cc:1196] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/065ca5b75c0d4f78b8c25e3ef9e56edf
I20260926 16:40:08.243975 24065 ts_tablet_manager.cc:1939] T 065ca5b75c0d4f78b8c25e3ef9e56edf P e23893e5125442d994690861b0ea4ae2: Deleting consensus metadata
I20260926 16:40:08.246032 23523 catalog_manager.cc:4841] TS e23893e5125442d994690861b0ea4ae2 (127.10.110.2:39389): tablet 065ca5b75c0d4f78b8c25e3ef9e56edf (table test-table [id=55bd32022b1d4a1e8acd4c95464d0286]) successfully deleted
I20260926 16:40:08.247164 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23575
I20260926 16:40:08.267630 24067 tablet_replica.cc:330] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: stopping tablet replica
I20260926 16:40:08.268215 24067 raft_consensus.cc:2229] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:08.268781 24067 raft_consensus.cc:2258] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:08.269649 24066 tablet_replica.cc:330] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: stopping tablet replica
I20260926 16:40:08.270282 24066 raft_consensus.cc:2229] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:40:08.271015 24066 raft_consensus.cc:2258] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:08.273380 24066 ts_tablet_manager.cc:1905] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:08.277803 24067 ts_tablet_manager.cc:1905] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:08.284245 24067 ts_tablet_manager.cc:1918] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:08.284456 24067 log.cc:1196] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/065ca5b75c0d4f78b8c25e3ef9e56edf
I20260926 16:40:08.285005 24067 ts_tablet_manager.cc:1939] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 86045369158249d6b9b43ab341a599d2: Deleting consensus metadata
I20260926 16:40:08.286779 24066 ts_tablet_manager.cc:1918] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:08.286975 24066 log.cc:1196] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/065ca5b75c0d4f78b8c25e3ef9e56edf
I20260926 16:40:08.287627 24066 ts_tablet_manager.cc:1939] T 065ca5b75c0d4f78b8c25e3ef9e56edf P 4924888ca3e8456d8b8bf0e8e0948995: Deleting consensus metadata
I20260926 16:40:08.288440 23524 catalog_manager.cc:4841] TS 86045369158249d6b9b43ab341a599d2 (127.10.110.1:42567): tablet 065ca5b75c0d4f78b8c25e3ef9e56edf (table test-table [id=55bd32022b1d4a1e8acd4c95464d0286]) successfully deleted
I20260926 16:40:08.289350 23524 catalog_manager.cc:4841] TS 4924888ca3e8456d8b8bf0e8e0948995 (127.10.110.3:33865): tablet 065ca5b75c0d4f78b8c25e3ef9e56edf (table test-table [id=55bd32022b1d4a1e8acd4c95464d0286]) successfully deleted
I20260926 16:40:08.295499 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.248147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.10.110.1:42567)
0926 16:40:08.248334 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:08.248341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:08.277948 (+ 29607us) server_negotiation.cc:407] Connection header received
0926 16:40:08.278256 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:08.278262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:08.278353 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:08.278483 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:08.280077 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.281220 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.283065 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.283788 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.283966 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.287771 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:08.287788 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:08.287808 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:08.287891 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:08.292229 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:08.293024 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:08.293034 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:08.293038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:08.293248 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:08.293730 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:08.293736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:08.293739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:08.294326 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:08.294658 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:08.294864 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 16:40:08.295116 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:40:08.296180 23730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:08.393932 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23575
I20260926 16:40:08.407980 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23739
I20260926 16:40:08.422791 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.408761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.10.110.2:39389)
0926 16:40:08.408945 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:08.408951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:08.408974 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:40:08.409084 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:08.409090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:08.409181 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:08.409298 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:08.410586 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.411699 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.413148 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.413780 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.413907 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.416322 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:08.416341 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:08.416362 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:08.416404 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:08.420085 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:08.420755 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:08.420764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:08.420768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:08.420932 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:08.421287 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:08.421291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:08.421295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:08.421722 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:08.422013 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:08.422208 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 16:40:08.422427 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:40:08.423527 23875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:08.511124 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23739
I20260926 16:40:08.525068 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23884
I20260926 16:40:08.542740 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:08.525913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.10.110.3:33865)
0926 16:40:08.526103 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:08.526110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:08.526206 (+ 96us) server_negotiation.cc:407] Connection header received
0926 16:40:08.526394 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:08.526399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:08.526492 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:08.526616 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:08.528191 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.529395 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.531226 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:08.531982 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:08.532120 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:08.535271 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:08.535289 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:08.535315 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:08.535371 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:08.539495 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:08.540391 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:08.540401 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:08.540404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:08.540595 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:08.541009 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:08.541015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:08.541018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:08.541502 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:08.541852 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:08.542091 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 16:40:08.542344 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:40:08.543496 24009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:08.665073 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23884
I20260926 16:40:08.681352 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 23505
I20260926 16:40:08.682500 23568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:08.792804 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 23505
I20260926 16:40:08.811743 10680 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:40:09.147421 10680 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:40:09.147662 10680 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:40:09.483382 10680 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:40:09.483 UTC [22557] LOG: received smart shutdown request
2026-09-26 16:40:09.486 UTC [22557] LOG: background worker "logical replication launcher" (PID 22566) exited with exit code 1
2026-09-26 16:40:09.486 UTC [22561] LOG: shutting down
2026-09-26 16:40:09.650 UTC [22557] LOG: database system is shut down
2026-09-26T16:40:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51875 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:40:09 dist-test-slave-whjp krb5kdc[24081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:40:09 dist-test-slave-whjp krb5kdc[24081](info): set up 2 sockets
Sep 26 16:40:09 dist-test-slave-whjp krb5kdc[24081](info): commencing operation
krb5kdc: starting...
W20260926 16:40:10.776856 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:40:10 dist-test-slave-whjp krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:40:10.995568 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38603
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:38603,127.10.110.61:40997,127.10.110.60:35629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:11.324473 24094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:11.324857 24094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:11.324995 24094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:11.332484 24094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:11.332617 24094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:11.332705 24094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:11.332779 24094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:11.332837 24094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:11.343045 24094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:38603,127.10.110.61:40997,127.10.110.60:35629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:11.344511 24094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:11.345850 24094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:11.357932 24099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:11.358337 24102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:11.357973 24100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:11.359949 24094 server_base.cc:1034] running on GCE node
I20260926 16:40:11.360961 24094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:11.362917 24094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:11.364269 24094 hybrid_clock.cc:648] HybridClock initialized: now 1790440811364181 us; error 77 us; skew 500 ppm
I20260926 16:40:11.365290 24094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:11 dist-test-slave-whjp krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:11.370090 24094 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:11.371951 24094 webserver.cc:460] Webserver started at http://127.10.110.62:39303/ using document root <none> and password file <none>
I20260926 16:40:11.372666 24094 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:11.372782 24094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:11.373049 24094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:11.376269 24094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "5fdcb1dc331c42e29c413686328667ab"
format_stamp: "Formatted at 2026-09-26 16:40:11 on dist-test-slave-whjp"
server_key: "ad4f90a680bf4415d32d70bfe6b2f8dc"
server_key_iv: "5326e6ba99f0efe6302f76c6cff1c522"
server_key_version: "encryptionkey@0"
I20260926 16:40:11.377089 24094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "5fdcb1dc331c42e29c413686328667ab"
format_stamp: "Formatted at 2026-09-26 16:40:11 on dist-test-slave-whjp"
server_key: "ad4f90a680bf4415d32d70bfe6b2f8dc"
server_key_iv: "5326e6ba99f0efe6302f76c6cff1c522"
server_key_version: "encryptionkey@0"
I20260926 16:40:11.382465 24094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:40:11.386381 24109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:11.387930 24094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:11.388099 24094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "5fdcb1dc331c42e29c413686328667ab"
format_stamp: "Formatted at 2026-09-26 16:40:11 on dist-test-slave-whjp"
server_key: "ad4f90a680bf4415d32d70bfe6b2f8dc"
server_key_iv: "5326e6ba99f0efe6302f76c6cff1c522"
server_key_version: "encryptionkey@0"
I20260926 16:40:11.388295 24094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:11.408733 24094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:11.412914 24094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:11.413161 24094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:11.430685 24094 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:38603
I20260926 16:40:11.430725 24160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:38603 every 8 connection(s)
I20260926 16:40:11.432111 24094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:11.435895 24161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:11.437176 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24094
I20260926 16:40:11.437399 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:40:11.438195 10680 external_mini_cluster.cc:1428] Setting key 8765ba8caa956e3ff9075a95cc98d2f6
I20260926 16:40:11.445446 24161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.110.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.61@KRBTEST.COM" created.
Sep 26 16:40:11 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:11.470374 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.449197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.10.110.62:38603)
0926 16:40:11.450039 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:11.450055 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:11.450523 (+ 468us) server_negotiation.cc:407] Connection header received
0926 16:40:11.452610 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:11.452615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:11.452853 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:11.453035 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:11.454296 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.455604 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.457059 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.457627 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.457817 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:11.462437 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:11.462449 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:11.462469 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:11.462513 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:11.466029 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:11.466845 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:11.466853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:11.466857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:11.467056 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:11.467443 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:11.467448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:11.467451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:11.468162 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:11.468359 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:11.469005 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 16:40:11.469276 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":389,"threads_started":1}
I20260926 16:40:11.470398 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.449201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:38603 (local address 127.0.0.1:39774)
0926 16:40:11.450365 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.450406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:11.452249 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:11.453063 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:11.453070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:11.453774 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:11.454111 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:11.454129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.455749 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.455752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:11.456923 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:11.456936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.457859 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.457866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:11.458033 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:11.458629 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:11.458662 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:11.462162 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:11.466220 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:11.466232 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:11.466243 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:11.466676 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:11.467175 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:11.467179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:11.467181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:11.467343 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:11.468293 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:11.468299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:11.468575 (+ 276us) client_negotiation.cc:769] Sending connection context
0926 16:40:11.468858 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 16:40:11.469445 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":63,"spinlock_wait_cycles":18176,"thread_start_us":734,"threads_started":1}
I20260926 16:40:11.473294 24161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:40:11.474434 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.473961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59848)
0926 16:40:11.474153 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.474285 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 16:40:11.474730 24110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
I20260926 16:40:11.476384 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.475992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59850)
0926 16:40:11.476147 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.476229 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:40:11.476606 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:40997: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab.
Entry for principal kudu/127.10.110.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab.
I20260926 16:40:11.492662 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:40997
--webserver_interface=127.10.110.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab
--principal=kudu/127.10.110.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:38603,127.10.110.61:40997,127.10.110.60:35629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:11.525993 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } attempt: 1
I20260926 16:40:11.527362 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.526726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59862)
0926 16:40:11.526981 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.527163 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 16:40:11.528728 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.528142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59868)
0926 16:40:11.528428 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.528529 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260926 16:40:11.528975 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:40997: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
I20260926 16:40:11.597235 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } attempt: 2
I20260926 16:40:11.598752 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.598044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59872)
0926 16:40:11.598446 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.598562 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:40:11.600296 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.599793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59882)
0926 16:40:11.600041 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.600126 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260926 16:40:11.600636 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:40997: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
I20260926 16:40:11.691905 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } attempt: 3
I20260926 16:40:11.693105 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.692560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59894)
0926 16:40:11.692793 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.692925 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 16:40:11.694170 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.693728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59902)
0926 16:40:11.693890 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.693995 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:40:11.694399 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:40997: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
I20260926 16:40:11.837687 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } attempt: 4
I20260926 16:40:11.838956 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.838347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59916)
0926 16:40:11.838667 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.838784 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 16:40:11.840117 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.839675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59920)
0926 16:40:11.839861 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.839962 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:40:11.840309 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:40997: Network error: Client connection negotiation failed: client connection to 127.10.110.61:40997: connect: Connection refused (error 111)
W20260926 16:40:11.848266 24168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:11.848526 24168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:11.848618 24168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:11.855741 24168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:11.855866 24168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:11.855931 24168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:11.855988 24168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:11.856029 24168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:11.865849 24168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:38603,127.10.110.61:40997,127.10.110.60:35629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab
--principal=kudu/127.10.110.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:40997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.110.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:11.867066 24168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:11.868359 24168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:11.879503 24173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:11.879586 24174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:11.879530 24176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:11.880698 24168 server_base.cc:1034] running on GCE node
I20260926 16:40:11.881490 24168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:11.882927 24168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:11.884136 24168 hybrid_clock.cc:648] HybridClock initialized: now 1790440811884092 us; error 33 us; skew 500 ppm
I20260926 16:40:11.884461 24168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:11 dist-test-slave-whjp krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:11.887463 24168 init.cc:377] Logged in from keytab as kudu/127.10.110.61@KRBTEST.COM (short username kudu)
I20260926 16:40:11.888965 24168 webserver.cc:460] Webserver started at http://127.10.110.61:38475/ using document root <none> and password file <none>
I20260926 16:40:11.889634 24168 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:11.889747 24168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:11.890025 24168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:11.893242 24168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data/instance:
uuid: "9d539d7f4a51438bb2afdb3a511bf6b0"
format_stamp: "Formatted at 2026-09-26 16:40:11 on dist-test-slave-whjp"
server_key: "73a347793485b524e29739af5d0fb992"
server_key_iv: "09c5d25156a7e3a4f49a65a4092df900"
server_key_version: "encryptionkey@0"
I20260926 16:40:11.894057 24168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal/instance:
uuid: "9d539d7f4a51438bb2afdb3a511bf6b0"
format_stamp: "Formatted at 2026-09-26 16:40:11 on dist-test-slave-whjp"
server_key: "73a347793485b524e29739af5d0fb992"
server_key_iv: "09c5d25156a7e3a4f49a65a4092df900"
server_key_version: "encryptionkey@0"
I20260926 16:40:11.899166 24168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:40:11.902619 24183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:11.903976 24168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:40:11.904145 24168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal
uuid: "9d539d7f4a51438bb2afdb3a511bf6b0"
format_stamp: "Formatted at 2026-09-26 16:40:11 on dist-test-slave-whjp"
server_key: "73a347793485b524e29739af5d0fb992"
server_key_iv: "09c5d25156a7e3a4f49a65a4092df900"
server_key_version: "encryptionkey@0"
I20260926 16:40:11.904325 24168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:11.918527 24168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:11.923007 24168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:11.923278 24168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:11.940820 24168 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.61:40997
I20260926 16:40:11.940863 24234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.61:40997 every 8 connection(s)
I20260926 16:40:11.942158 24168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
I20260926 16:40:11.945993 24235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:11.946056 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24168
I20260926 16:40:11.946229 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-1/wal/instance
I20260926 16:40:11.946754 10680 external_mini_cluster.cc:1428] Setting key 59896d531eaf9f0ec8bd1385772593b8
I20260926 16:40:11.955327 24235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.110.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.60@KRBTEST.COM" created.
Sep 26 16:40:11 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:11.979287 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.958299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.10.110.62:38603)
0926 16:40:11.958465 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:11.958470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:11.960028 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 16:40:11.962013 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:11.962020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:11.962114 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:11.962202 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:11.963871 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.964570 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.966184 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.966609 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.966735 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:11.971992 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:11.972000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:11.972004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:11.972053 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:11.975447 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:11.976470 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:11.976476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:11.976480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:11.976651 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:11.977224 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:11.977230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:11.977233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:11.977623 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:11.977789 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:11.978821 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0926 16:40:11.979025 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:40:11.980883 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.958946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:38603 (local address 127.0.0.1:39782)
0926 16:40:11.959804 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.959887 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:11.961712 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:11.962464 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:11.962474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:11.963023 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:11.963637 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:11.963658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.964730 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.964735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:11.966020 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:11.966034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.966847 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.966855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:11.967080 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:11.967912 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:11.967938 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:11.971628 (+ 3690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:11.975681 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:11.975691 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:11.975708 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:11.976324 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:11.976796 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:11.976801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:11.976803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:11.977034 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:11.977784 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:11.977793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:11.978325 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 16:40:11.978712 (+ 387us) client_negotiation.cc:240] Negotiation successful
0926 16:40:11.979427 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":351,"threads_started":1}
I20260926 16:40:11.982735 24235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:40:11 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:40:11.997776 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.983624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59926)
0926 16:40:11.983810 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:11.983825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:11.983950 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:11.985068 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:11.985072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:11.985534 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:11.985685 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:11.985694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.987106 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.987112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:11.988094 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:11.988108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.988929 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.988934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:11.989168 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:11.989716 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:11.989733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:11.991691 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:11.994983 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:11.994987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:11.994990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:11.995505 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:11.995948 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:11.995968 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:11.995972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:11.996185 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:11.996827 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:11.996832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:11.996967 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 16:40:11.997086 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 16:40:11.997389 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:40:11.997880 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:11.983958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.10.110.61:40997)
0926 16:40:11.984611 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:11.984617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:11.984644 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:40:11.984699 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:11.984705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:11.984894 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:11.985074 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:11.985836 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.986896 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.988232 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:11.988790 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:11.988899 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:11.991945 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:11.991958 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:11.991976 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:11.992015 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:11.994842 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:11.995645 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:11.995652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:11.995655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:11.995782 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:11.996304 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:11.996308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:11.996310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:11.996670 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:11.996893 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:11.997251 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:40:11.997507 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":504,"thread_start_us":373,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab.
Entry for principal kudu/127.10.110.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab.
I20260926 16:40:11.999513 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:35629
--webserver_interface=127.10.110.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab
--principal=kudu/127.10.110.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:38603,127.10.110.61:40997,127.10.110.60:35629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:12.000520 24235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:40:12.001613 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.001106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59920)
0926 16:40:12.001283 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.001422 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:40:12.001849 24187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.003357 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.002991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59930)
0926 16:40:12.003126 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.003230 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 16:40:12.003587 24235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.052924 24235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 1
I20260926 16:40:12.054155 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.053670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59936)
0926 16:40:12.053866 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.053985 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 16:40:12.055349 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.054920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59950)
0926 16:40:12.055115 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.055220 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:40:12.058804 24235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.127178 24235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 2
I20260926 16:40:12.128429 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.127953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59964)
0926 16:40:12.128182 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.128276 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 16:40:12.129490 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.129115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59968)
0926 16:40:12.129299 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.129373 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 16:40:12.129760 24235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.139573 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } attempt: 5
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:40:12.154170 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.140300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59932)
0926 16:40:12.140649 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.140675 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:12.140797 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:12.141240 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:12.141244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:12.141594 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:12.141779 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.141788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.143210 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.143215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.144148 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.144162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.145132 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.145138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.145296 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.145914 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:12.145934 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:12.148216 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:12.151765 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.151769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.151772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.152170 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.152553 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.152556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.152558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.152690 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.153358 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:12.153363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:12.153533 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 16:40:12.153690 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:40:12.153923 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260926 16:40:12.154366 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.140539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.10.110.61:40997)
0926 16:40:12.140756 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.140763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.140789 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:40:12.140928 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.140933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.141017 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.141181 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.141947 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.143011 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.144285 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.144935 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.145071 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.148418 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.148424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.148429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.148468 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.151609 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.152285 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.152291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.152295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.152450 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.152799 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.152804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.152807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.153195 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.153361 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.153835 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.154069 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 16:40:12.155403 24161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:40:12.156373 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.156018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59984)
0926 16:40:12.156177 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.156248 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 16:40:12.157207 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.156883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:59996)
0926 16:40:12.157046 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.157107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:40:12.157375 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.208576 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 1
I20260926 16:40:12.209703 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.209218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60000)
0926 16:40:12.209426 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.209521 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 16:40:12.210664 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.210327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60006)
0926 16:40:12.210472 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.210559 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 16:40:12.210875 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.220963 24235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 3
I20260926 16:40:12.222172 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.221685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60020)
0926 16:40:12.221914 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.222013 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 16:40:12.223162 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.222760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60034)
0926 16:40:12.222969 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.223044 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260926 16:40:12.223357 24235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.279103 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 2
I20260926 16:40:12.280216 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.279712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60036)
0926 16:40:12.279961 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.280045 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 16:40:12.281111 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.280785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60044)
0926 16:40:12.280926 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.281000 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 16:40:12.281296 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
W20260926 16:40:12.337056 24242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:12.337340 24242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:12.337469 24242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:12.344722 24242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:12.344841 24242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:12.344923 24242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:12.344995 24242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:12.345052 24242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:12.354841 24242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:38603,127.10.110.61:40997,127.10.110.60:35629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab
--principal=kudu/127.10.110.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:35629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.110.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:12.356158 24242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:12.357402 24242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:40:12.366536 24235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 4
I20260926 16:40:12.367664 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.367185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60048)
0926 16:40:12.367406 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.367509 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:40:12.368602 24250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:12.368630 24248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:12.368919 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.368467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60064)
0926 16:40:12.368686 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.368776 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260926 16:40:12.369231 24235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
W20260926 16:40:12.369343 24247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:12.369799 24242 server_base.cc:1034] running on GCE node
I20260926 16:40:12.370700 24242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:12.372385 24242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:12.373584 24242 hybrid_clock.cc:648] HybridClock initialized: now 1790440812373529 us; error 41 us; skew 500 ppm
I20260926 16:40:12.373974 24242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440812, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:12.377681 24242 init.cc:377] Logged in from keytab as kudu/127.10.110.60@KRBTEST.COM (short username kudu)
I20260926 16:40:12.379027 24242 webserver.cc:460] Webserver started at http://127.10.110.60:32879/ using document root <none> and password file <none>
I20260926 16:40:12.379684 24242 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:12.379788 24242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:12.380092 24242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:12.382897 24242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data/instance:
uuid: "042c4d81fc764a23a97a39d998f04b21"
format_stamp: "Formatted at 2026-09-26 16:40:12 on dist-test-slave-whjp"
server_key: "da6e70cfae126ac9055f4be560f064dd"
server_key_iv: "9d890aa5a99b3a885a0274288576ed50"
server_key_version: "encryptionkey@0"
I20260926 16:40:12.383649 24242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal/instance:
uuid: "042c4d81fc764a23a97a39d998f04b21"
format_stamp: "Formatted at 2026-09-26 16:40:12 on dist-test-slave-whjp"
server_key: "da6e70cfae126ac9055f4be560f064dd"
server_key_iv: "9d890aa5a99b3a885a0274288576ed50"
server_key_version: "encryptionkey@0"
I20260926 16:40:12.387465 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 3
I20260926 16:40:12.388484 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.388037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60076)
0926 16:40:12.388259 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.388339 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:40:12.388505 24242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:40:12.389441 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.389117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60080)
0926 16:40:12.389276 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.389347 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 16:40:12.389624 24161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:35629: Network error: Client connection negotiation failed: client connection to 127.10.110.60:35629: connect: Connection refused (error 111)
I20260926 16:40:12.392045 24257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:12.393314 24242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:12.393436 24242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal
uuid: "042c4d81fc764a23a97a39d998f04b21"
format_stamp: "Formatted at 2026-09-26 16:40:12 on dist-test-slave-whjp"
server_key: "da6e70cfae126ac9055f4be560f064dd"
server_key_iv: "9d890aa5a99b3a885a0274288576ed50"
server_key_version: "encryptionkey@0"
I20260926 16:40:12.393589 24242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:12.424086 24242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:12.427739 24242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:12.427985 24242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:12.445454 24242 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.60:35629
I20260926 16:40:12.445479 24308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.60:35629 every 8 connection(s)
I20260926 16:40:12.446782 24242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
I20260926 16:40:12.450322 24309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:12.450892 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24242
I20260926 16:40:12.451067 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440615325743-10680-0/minicluster-data/master-2/wal/instance
I20260926 16:40:12.451498 10680 external_mini_cluster.cc:1428] Setting key f0445ae5843840e32f7561cf4ada4ef7
I20260926 16:40:12.458951 24309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440812, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:12.473562 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.453290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.10.110.60:35629)
0926 16:40:12.454051 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.454062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.454101 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:40:12.455959 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.455982 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.456387 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.456724 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.458521 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.459934 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.461692 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.462630 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.462785 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.466718 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.466726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.466738 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.466776 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.469771 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.470411 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.470420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.470429 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.470568 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.470911 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.470916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.470920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.471422 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.471673 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.472171 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.472602 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":358,"threads_started":1}
I20260926 16:40:12.474694 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.461661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.10.110.62:38603)
0926 16:40:12.461855 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.461861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.462287 (+ 426us) server_negotiation.cc:407] Connection header received
0926 16:40:12.462455 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.462459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.462522 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.462600 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.463752 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.464464 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.465701 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.466195 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.466302 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.469725 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.469730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.469733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.469766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.472225 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.472943 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.472949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.472952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.473074 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.473399 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.473404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.473407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.473646 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.473782 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.474273 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.474431 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:40:12.474879 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.461577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:38603 (local address 127.0.0.1:39788)
0926 16:40:12.462157 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.462199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:12.462342 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:12.462679 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:12.462685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:12.463274 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:12.463589 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.463600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.464604 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.464608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.465569 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.465580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.466405 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.466412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.466543 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.467119 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:12.467138 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:12.469545 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:12.472340 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.472351 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.472358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.472838 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.473184 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.473189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.473193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.473306 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.473804 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:12.473811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:12.474007 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 16:40:12.474205 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 16:40:12.474503 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260926 16:40:12.477108 24309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440812, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:12.490886 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.477804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:40997 (local address 127.0.0.1:59948)
0926 16:40:12.477974 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.477991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:12.478094 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:12.478379 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:12.478383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:12.478745 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:12.478876 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.478884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.480129 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.480142 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.481371 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.481383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.482245 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.482251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.482408 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.482812 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:12.482834 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:12.485127 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:12.488578 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.488582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.488585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.488963 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.489348 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.489353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.489357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.489493 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.490063 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:12.490068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:12.490252 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 16:40:12.490386 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 16:40:12.490591 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 16:40:12.491192 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.477830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.10.110.61:40997)
0926 16:40:12.477999 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.478003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.478040 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:40:12.478203 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.478206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.478264 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.478337 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.479009 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.480001 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.481509 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.482092 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.482198 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.485339 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.485344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.485348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.485379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.488438 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.489082 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.489088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.489092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.489257 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.489592 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.489598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.489602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.489941 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.490080 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.490497 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.490717 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:40:12.491852 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.476590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.10.110.62:38603)
0926 16:40:12.476742 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.476747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.476832 (+ 85us) server_negotiation.cc:407] Connection header received
0926 16:40:12.476996 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.477000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.477059 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.477132 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.478623 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.479529 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.481252 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.481780 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.481909 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.486151 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.486157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.486161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.486197 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.489162 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.489782 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.489786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.489789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.489928 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.490305 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.490308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.490310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.490620 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.490933 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.491334 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.491549 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:40:12.492169 24309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440812, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:40:12.504393 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.492860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60104)
0926 16:40:12.493004 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.493021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:12.493112 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:12.493479 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:12.493484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:12.493860 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:12.494007 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.494017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.494891 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.494896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.495906 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.495918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.496667 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.496672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.496835 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.497423 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:12.497438 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:12.499582 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:12.502237 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.502242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.502247 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.502554 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.502848 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.502853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.502857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.502974 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.503430 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:12.503434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:12.503561 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 16:40:12.503695 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 16:40:12.503994 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 16:40:12.504429 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.493054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.10.110.60:35629)
0926 16:40:12.493221 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.493228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.493244 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:40:12.493288 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.493293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.493356 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.493447 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.494116 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.494772 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.496029 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.496534 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.496636 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.499747 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.499752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.499755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.499781 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.502126 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.502649 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.502653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.502657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.502760 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.503062 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.503066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.503070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.503329 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.503447 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.503784 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.504152 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":39}
I20260926 16:40:12.507112 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.493675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.10.110.61:40997)
0926 16:40:12.493808 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.493812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.493824 (+ 12us) server_negotiation.cc:407] Connection header received
0926 16:40:12.493859 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.493862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.493908 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.493975 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.495254 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.495985 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.497214 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.497709 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.497811 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.502074 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.502078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.502083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.502120 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.504722 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.505485 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.505491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.505494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.505642 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.505980 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.505983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.505985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.506223 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.506412 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.506702 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.506857 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 16:40:12.510001 24309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: Bootstrap starting.
I20260926 16:40:12.511785 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:40:12.513583 24309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:12.514822 24309 log.cc:824] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:12.517992 24309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: No bootstrap required, opened a new log
I20260926 16:40:12.528159 24309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.528602 24309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:12.528731 24309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042c4d81fc764a23a97a39d998f04b21, State: Initialized, Role: FOLLOWER
I20260926 16:40:12.529505 24309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.533696 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.534119 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.535070 24309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:12.536410 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.517680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.10.110.61:40997)
0926 16:40:12.517840 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.517845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.518709 (+ 864us) server_negotiation.cc:407] Connection header received
0926 16:40:12.518955 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.518960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.519031 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.519098 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.521438 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.522573 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.524853 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.525512 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.525603 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.529065 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.529071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.529077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.529112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.532786 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.533844 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.533850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.533853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.534004 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.534658 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.534663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.534666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.535022 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.535260 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.535815 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.536065 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:40:12.536844 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.517300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.10.110.62:38603)
0926 16:40:12.517492 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.517498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.518004 (+ 506us) server_negotiation.cc:407] Connection header received
0926 16:40:12.518310 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.518316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.518402 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.518502 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.520802 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.521883 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.523994 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.524673 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.524768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.528968 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.528975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.528979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.529031 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.532538 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.534579 (+ 2041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.534585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.534589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.534752 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.535176 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.535181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.535184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.535507 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.535721 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.536284 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.536511 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:40:12.542152 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.518807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.10.110.60:35629)
0926 16:40:12.518981 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.518987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.525701 (+ 6714us) server_negotiation.cc:407] Connection header received
0926 16:40:12.526096 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.526102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.526179 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.526247 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.528366 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.529401 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.531190 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.531896 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.531989 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.535080 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.535086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.535090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.535127 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.538654 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.539753 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.539762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.539766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.540027 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.540436 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.540441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.540443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.540862 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.541074 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.541432 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.541607 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 16:40:12.549889 24333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:40:12.550000 24333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:40:12.546586 24242 master_runner.cc:186] Error getting master registration for 127.10.110.62:38603: OK, instance_id { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" instance_seqno: 1790440811425545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:40:12.556465 24242 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:40:12.566860 24161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 4
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
I20260926 16:40:12.581557 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.567627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.10.110.60:35629)
0926 16:40:12.567804 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.567810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.567851 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:40:12.567901 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.567904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.567974 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.568051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.568811 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.569589 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.571332 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.571807 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.571951 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.575094 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.575101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.575107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.575158 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.578602 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.579450 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.579456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.579460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.579632 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.580070 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.580074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.580077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.580434 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.580542 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.581082 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.581271 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:40:12.581800 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.567438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60128)
0926 16:40:12.567605 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.567644 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:12.567749 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:12.568098 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:12.568103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:12.568500 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:12.568669 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.568679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.569741 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.569747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.571197 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.571213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.571967 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.571973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.572158 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.572706 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:12.572723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:12.574905 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:12.578780 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.578786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.578790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.579308 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.579764 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.579770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.579773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.579958 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.580565 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:12.580570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:12.580765 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 16:40:12.580925 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 16:40:12.581239 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:40:12.586274 24161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: Bootstrap starting.
I20260926 16:40:12.589437 24161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:12.590816 24161 log.cc:824] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:12.594466 24161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: No bootstrap required, opened a new log
I20260926 16:40:12.601761 24161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.602181 24161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:12.602303 24161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdcb1dc331c42e29c413686328667ab, State: Initialized, Role: FOLLOWER
I20260926 16:40:12.602931 24161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.603948 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.604331 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.605001 24161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:40:12.614280 24347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:40:12.614445 24347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:40:12.613543 24094 master_runner.cc:186] Error getting master registration for 127.10.110.61:40997: OK, instance_id { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" instance_seqno: 1790440811935749 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:40:12.621826 24094 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:40:12.665906 24115 tablet.cc:2369] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:12.668531 24235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } attempt: 5
Sep 26 16:40:12 dist-test-slave-whjp krb5kdc[24081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440811, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
W20260926 16:40:12.679949 24263 tablet.cc:2369] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:12.682276 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.669102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:35629 (local address 127.0.0.1:60136)
0926 16:40:12.669355 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:12.669373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:12.669490 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:12.669806 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:12.669819 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:12.670239 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:12.670397 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.670405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.671433 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.671439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.672448 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:12.672461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.673168 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.673174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:12.673366 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.673928 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:12.673946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:12.676061 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:12.679490 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.679494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.679498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.680024 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.680546 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:12.680551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:12.680565 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:12.680725 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:12.681437 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:12.681442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:12.681615 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 16:40:12.681768 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 16:40:12.682019 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 16:40:12.683077 24312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:12.669182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.10.110.60:35629)
0926 16:40:12.669359 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:12.669368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:12.669428 (+ 60us) server_negotiation.cc:407] Connection header received
0926 16:40:12.669609 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:12.669613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:12.669676 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:12.669760 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:12.670547 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.671282 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.672602 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:12.673026 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:12.673152 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:12.676253 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:12.676258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:12.676261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:12.676294 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:12.679333 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.680271 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.680276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.680279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.680419 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:12.680847 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:12.680852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:12.680855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:12.681207 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:12.681378 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:12.682619 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0926 16:40:12.682811 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:40:12.687332 24235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: Bootstrap starting.
I20260926 16:40:12.690362 24235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:12.691346 24235 log.cc:824] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:12.694104 24235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: No bootstrap required, opened a new log
I20260926 16:40:12.705423 24235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.705804 24235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:12.705921 24235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d539d7f4a51438bb2afdb3a511bf6b0, State: Initialized, Role: FOLLOWER
I20260926 16:40:12.706614 24235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.707640 24350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.708122 24350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.708992 24235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:12.734531 24168 master_runner.cc:263] No leader master found from master 9d539d7f4a51438bb2afdb3a511bf6b0
I20260926 16:40:12.734683 24168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:40:12.737907 24361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:40:12.738030 24361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:40:12.740787 24322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:12.741115 24322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.742805 24322 leader_election.cc:288] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fdcb1dc331c42e29c413686328667ab (127.10.110.62:38603), 9d539d7f4a51438bb2afdb3a511bf6b0 (127.10.110.61:40997)
I20260926 16:40:12.743530 24136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "042c4d81fc764a23a97a39d998f04b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fdcb1dc331c42e29c413686328667ab" is_pre_election: true
I20260926 16:40:12.744130 24136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042c4d81fc764a23a97a39d998f04b21 in term 0.
I20260926 16:40:12.745114 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "042c4d81fc764a23a97a39d998f04b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" is_pre_election: true
I20260926 16:40:12.745666 24210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 042c4d81fc764a23a97a39d998f04b21 in term 0.
I20260926 16:40:12.746369 24258 leader_election.cc:302] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042c4d81fc764a23a97a39d998f04b21, 9d539d7f4a51438bb2afdb3a511bf6b0; no voters:
I20260926 16:40:12.747398 24322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:40:12.747552 24322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:40:12.747653 24322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:12.750948 24322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.752224 24136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "042c4d81fc764a23a97a39d998f04b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fdcb1dc331c42e29c413686328667ab"
I20260926 16:40:12.752453 24136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:12.756028 24136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042c4d81fc764a23a97a39d998f04b21 in term 1.
I20260926 16:40:12.758065 24210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "042c4d81fc764a23a97a39d998f04b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0"
I20260926 16:40:12.758314 24210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:12.763300 24210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 042c4d81fc764a23a97a39d998f04b21 in term 1.
I20260926 16:40:12.765568 24258 leader_election.cc:302] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 042c4d81fc764a23a97a39d998f04b21, 9d539d7f4a51438bb2afdb3a511bf6b0; no voters:
I20260926 16:40:12.767082 24322 leader_election.cc:288] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [CANDIDATE]: Term 1 election: Requested vote from peers 5fdcb1dc331c42e29c413686328667ab (127.10.110.62:38603), 9d539d7f4a51438bb2afdb3a511bf6b0 (127.10.110.61:40997)
I20260926 16:40:12.769309 24322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:12.770130 24322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [term 1 LEADER]: Becoming Leader. State: Replica: 042c4d81fc764a23a97a39d998f04b21, State: Running, Role: LEADER
I20260926 16:40:12.771003 24322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } }
I20260926 16:40:12.780260 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042c4d81fc764a23a97a39d998f04b21. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.781261 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:12.800397 24365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:12.802132 24365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:12.821159 24210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [term 1 FOLLOWER]: Refusing update from remote peer 042c4d81fc764a23a97a39d998f04b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:12.821255 24136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [term 1 FOLLOWER]: Refusing update from remote peer 042c4d81fc764a23a97a39d998f04b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:12.822206 24363 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:12.822722 24322 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:12.864225 24350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042c4d81fc764a23a97a39d998f04b21. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.864557 24350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.868119 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042c4d81fc764a23a97a39d998f04b21. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.868449 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.870071 24366 mvcc.cc:205] Tried to move back new op lower bound from 7333645569303416832 to 7333645569122553856. Current Snapshot: MvccSnapshot[applied={T|T < 7333645569303416832}]
I20260926 16:40:12.877990 24367 mvcc.cc:205] Tried to move back new op lower bound from 7333645569303416832 to 7333645569122553856. Current Snapshot: MvccSnapshot[applied={T|T < 7333645569303416832}]
I20260926 16:40:12.885880 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.886166 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.890033 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.890398 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:12.890723 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.890980 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:12.891681 24350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "042c4d81fc764a23a97a39d998f04b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fdcb1dc331c42e29c413686328667ab" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38603 } } peers { permanent_uuid: "9d539d7f4a51438bb2afdb3a511bf6b0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 40997 } } peers { permanent_uuid: "042c4d81fc764a23a97a39d998f04b21" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 35629 } } }
I20260926 16:40:12.891968 24350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:40:12.904374 24365 catalog_manager.cc:1347] Generated new cluster ID: 7e000a2f30b74a27a194e153193c78d1
I20260926 16:40:12.904492 24365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:12.934546 24365 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:40:12.936405 24365 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:12.963164 24365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 042c4d81fc764a23a97a39d998f04b21: Generated new TSK 0
I20260926 16:40:12.964329 24365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:40:13.616555 24347 catalog_manager.cc:1259] Loaded cluster ID: 7e000a2f30b74a27a194e153193c78d1
I20260926 16:40:13.616662 24347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: loading cluster ID for follower catalog manager: success
I20260926 16:40:13.619042 24347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: acquiring CA information for follower catalog manager: success
I20260926 16:40:13.620833 24347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fdcb1dc331c42e29c413686328667ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:40:13.739950 24361 catalog_manager.cc:1259] Loaded cluster ID: 7e000a2f30b74a27a194e153193c78d1
I20260926 16:40:13.740052 24361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: loading cluster ID for follower catalog manager: success
I20260926 16:40:13.742539 24361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: acquiring CA information for follower catalog manager: success
I20260926 16:40:13.744262 24361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d539d7f4a51438bb2afdb3a511bf6b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:40:14.078598 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24094
I20260926 16:40:14.079674 24156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:14.188112 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24094
I20260926 16:40:14.204996 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24168
I20260926 16:40:14.206050 24230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:14.309456 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24168
I20260926 16:40:14.325397 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24242
I20260926 16:40:14.326478 24304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:14.442727 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24242
2026-09-26T16:40:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4688 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:40:14 dist-test-slave-whjp krb5kdc[24385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:40:14 dist-test-slave-whjp krb5kdc[24385](info): set up 2 sockets
Sep 26 16:40:14 dist-test-slave-whjp krb5kdc[24385](info): commencing operation
krb5kdc: starting...
W20260926 16:40:15.452848 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:40:15 dist-test-slave-whjp krb5kdc[24385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:40:15.665109 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40049
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:41083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:15.976527 24398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:15.976809 24398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:15.976945 24398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:15.984102 24398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:15.984207 24398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:15.984274 24398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:15.984355 24398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:15.984416 24398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:15.994230 24398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:41083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:40049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:15.995568 24398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:15.996881 24398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:16.008203 24403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:16.008215 24406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:16.008172 24404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:16.008901 24398 server_base.cc:1034] running on GCE node
I20260926 16:40:16.009618 24398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:16.010985 24398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:16.012168 24398 hybrid_clock.cc:648] HybridClock initialized: now 1790440816012113 us; error 41 us; skew 500 ppm
I20260926 16:40:16.012465 24398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:16 dist-test-slave-whjp krb5kdc[24385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440816, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:16.018862 24398 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:16.020323 24398 webserver.cc:460] Webserver started at http://127.10.110.62:42183/ using document root <none> and password file <none>
I20260926 16:40:16.020960 24398 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:16.021057 24398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:16.021313 24398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:16.024248 24398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "e7a7ba79080a467585c4449d3d15bd2f"
format_stamp: "Formatted at 2026-09-26 16:40:16 on dist-test-slave-whjp"
server_key: "fdec6fc1f54debbb1dfab1cda79356ab"
server_key_iv: "0b3f12f0fd2003e8b9648f9973166e54"
server_key_version: "encryptionkey@0"
I20260926 16:40:16.024936 24398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "e7a7ba79080a467585c4449d3d15bd2f"
format_stamp: "Formatted at 2026-09-26 16:40:16 on dist-test-slave-whjp"
server_key: "fdec6fc1f54debbb1dfab1cda79356ab"
server_key_iv: "0b3f12f0fd2003e8b9648f9973166e54"
server_key_version: "encryptionkey@0"
I20260926 16:40:16.029874 24398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:40:16.033289 24413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:16.034708 24398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:16.034936 24398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "e7a7ba79080a467585c4449d3d15bd2f"
format_stamp: "Formatted at 2026-09-26 16:40:16 on dist-test-slave-whjp"
server_key: "fdec6fc1f54debbb1dfab1cda79356ab"
server_key_iv: "0b3f12f0fd2003e8b9648f9973166e54"
server_key_version: "encryptionkey@0"
I20260926 16:40:16.035140 24398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:16.048153 24398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:16.052386 24398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:16.052639 24398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:16.074759 24398 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:40049
I20260926 16:40:16.074781 24464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:40049 every 8 connection(s)
I20260926 16:40:16.076247 24398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:16.080158 24465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:16.085902 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24398
I20260926 16:40:16.086123 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:40:16.086946 10680 external_mini_cluster.cc:1428] Setting key d7c645ebdf67c19137d09be78db97c81
I20260926 16:40:16.095336 24465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f: Bootstrap starting.
I20260926 16:40:16.099113 24465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:16.100442 24465 log.cc:824] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:16.104117 24465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f: No bootstrap required, opened a new log
Sep 26 16:40:16 dist-test-slave-whjp krb5kdc[24385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:16.113401 24465 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a7ba79080a467585c4449d3d15bd2f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40049 } }
I20260926 16:40:16.113781 24465 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:16.113886 24465 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a7ba79080a467585c4449d3d15bd2f, State: Initialized, Role: FOLLOWER
I20260926 16:40:16.114552 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:16.090179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56352 (local address 127.10.110.62:40049)
0926 16:40:16.091608 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:16.091622 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:16.091670 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:40:16.094682 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:16.094711 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:16.095173 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:16.095610 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:16.097558 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:16.098902 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:16.100720 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:16.101772 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:16.101943 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:16.107055 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:16.107067 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:16.107083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:16.107126 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:16.110198 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:16.111114 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:16.111121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:16.111127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:16.111315 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:16.111756 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:16.111759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:16.111761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:16.112378 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:16.112590 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:16.113021 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 16:40:16.113386 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1184,"thread_start_us":394,"threads_started":1}
I20260926 16:40:16.114499 24465 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a7ba79080a467585c4449d3d15bd2f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40049 } }
I20260926 16:40:16.114768 24465 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:16.114938 24465 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:16.115144 24465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:16.117873 24465 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a7ba79080a467585c4449d3d15bd2f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40049 } }
I20260926 16:40:16.118383 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:40:16.118397 24465 leader_election.cc:302] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a7ba79080a467585c4449d3d15bd2f; no voters:
I20260926 16:40:16.118970 24465 leader_election.cc:288] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:40:16.119119 24470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:16.120855 24465 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:16.121376 24470 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [term 1 LEADER]: Becoming Leader. State: Replica: e7a7ba79080a467585c4449d3d15bd2f, State: Running, Role: LEADER
I20260926 16:40:16.122008 24470 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a7ba79080a467585c4449d3d15bd2f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40049 } }
I20260926 16:40:16.125195 24472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a7ba79080a467585c4449d3d15bd2f. Latest consensus state: current_term: 1 leader_uuid: "e7a7ba79080a467585c4449d3d15bd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a7ba79080a467585c4449d3d15bd2f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40049 } } }
I20260926 16:40:16.125168 24471 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a7ba79080a467585c4449d3d15bd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a7ba79080a467585c4449d3d15bd2f" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40049 } } }
I20260926 16:40:16.125648 24472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:16.125685 24471 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:16.126950 24481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:16.131974 24481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:16.141358 24481 catalog_manager.cc:1347] Generated new cluster ID: a5b70543b05940e9bfb5a9d6d93298a2
I20260926 16:40:16.141474 24481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:16.149178 24481 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:40:16.150508 24481 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:16.161309 24481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a7ba79080a467585c4449d3d15bd2f: Generated new TSK 0
I20260926 16:40:16.162220 24481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:40:16.180240 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24398
I20260926 16:40:16.181535 24460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:16.271785 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24398
2026-09-26T16:40:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1823 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:40:16 dist-test-slave-whjp krb5kdc[24493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:40:16 dist-test-slave-whjp krb5kdc[24493](info): set up 2 sockets
Sep 26 16:40:16 dist-test-slave-whjp krb5kdc[24493](info): commencing operation
krb5kdc: starting...
W20260926 16:40:17.269241 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:40:17 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:40:17.490975 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36663
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:17.817934 24506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:17.818212 24506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:17.818348 24506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:17.825769 24506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:17.825919 24506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:17.826007 24506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:17.826090 24506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:17.826149 24506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:17.836115 24506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:17.837383 24506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:17.838599 24506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:17.850647 24511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:17.850607 24514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:17.850677 24512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:17.851534 24506 server_base.cc:1034] running on GCE node
I20260926 16:40:17.852418 24506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:17.853988 24506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:17.855233 24506 hybrid_clock.cc:648] HybridClock initialized: now 1790440817855143 us; error 63 us; skew 500 ppm
I20260926 16:40:17.855638 24506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:17 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440817, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:17.858814 24506 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:17.860337 24506 webserver.cc:460] Webserver started at http://127.10.110.62:34485/ using document root <none> and password file <none>
I20260926 16:40:17.861011 24506 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:17.861125 24506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:17.861409 24506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:17.864485 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "1731653a8faf450691ff92da350052d7"
format_stamp: "Formatted at 2026-09-26 16:40:17 on dist-test-slave-whjp"
server_key: "af32c31a5e10a1eff892944e0c403cd7"
server_key_iv: "252746bc97ee7c8e5d134d36b4fe509b"
server_key_version: "encryptionkey@0"
I20260926 16:40:17.865269 24506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "1731653a8faf450691ff92da350052d7"
format_stamp: "Formatted at 2026-09-26 16:40:17 on dist-test-slave-whjp"
server_key: "af32c31a5e10a1eff892944e0c403cd7"
server_key_iv: "252746bc97ee7c8e5d134d36b4fe509b"
server_key_version: "encryptionkey@0"
I20260926 16:40:17.870561 24506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:40:17.874114 24521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:17.875408 24506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:40:17.875568 24506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "1731653a8faf450691ff92da350052d7"
format_stamp: "Formatted at 2026-09-26 16:40:17 on dist-test-slave-whjp"
server_key: "af32c31a5e10a1eff892944e0c403cd7"
server_key_iv: "252746bc97ee7c8e5d134d36b4fe509b"
server_key_version: "encryptionkey@0"
I20260926 16:40:17.875771 24506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:17.891975 24506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:17.896235 24506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:17.896504 24506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:17.912745 24506 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:36663
I20260926 16:40:17.912767 24572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:36663 every 8 connection(s)
I20260926 16:40:17.914021 24506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:17.917867 24573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:17.922560 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24506
I20260926 16:40:17.922755 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:40:17.923403 10680 external_mini_cluster.cc:1428] Setting key 8518e930743a8bc5d2b8be64266a16fd
I20260926 16:40:17.930840 24573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap starting.
I20260926 16:40:17.934773 24573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:17.935959 24573 log.cc:824] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:17.939328 24573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: No bootstrap required, opened a new log
Sep 26 16:40:17 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:17.948478 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:17.925506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.10.110.62:36663)
0926 16:40:17.926430 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:17.926448 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:17.926512 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:40:17.929148 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:17.929181 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:17.929733 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:17.930215 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:17.931628 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:17.933720 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:17.935035 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:17.935764 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:17.935970 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:17.940979 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:17.940987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:17.941005 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:17.941044 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:17.944155 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:17.945044 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:17.945052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:17.945058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:17.945263 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:17.945718 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:17.945723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:17.945725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:17.946281 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:17.946517 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:17.946987 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 16:40:17.947362 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":367,"threads_started":1}
I20260926 16:40:17.948395 24573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:17.948815 24573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:17.948909 24573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1731653a8faf450691ff92da350052d7, State: Initialized, Role: FOLLOWER
I20260926 16:40:17.949430 24573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:17.949659 24573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:17.949869 24573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:17.950042 24573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:17.952386 24573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:17.952853 24573 leader_election.cc:302] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1731653a8faf450691ff92da350052d7; no voters:
I20260926 16:40:17.953372 24573 leader_election.cc:288] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:40:17.953493 24578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:17.954106 24578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 LEADER]: Becoming Leader. State: Replica: 1731653a8faf450691ff92da350052d7, State: Running, Role: LEADER
I20260926 16:40:17.954579 24578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:17.955253 24573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:17.957635 24580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1731653a8faf450691ff92da350052d7. Latest consensus state: current_term: 1 leader_uuid: "1731653a8faf450691ff92da350052d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } } }
I20260926 16:40:17.958032 24580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:17.958758 24579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1731653a8faf450691ff92da350052d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } } }
I20260926 16:40:17.959228 24579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:17.959674 24583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:17.965034 24583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:17.975008 24583 catalog_manager.cc:1347] Generated new cluster ID: 7a0f832428924585b7ddbd1c02f19757
I20260926 16:40:17.975137 24583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:17.989785 24583 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:40:17.991881 24583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:18.003605 24583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Generated new TSK 0
I20260926 16:40:18.004848 24583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:40:18.061396 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36663
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:18.416121 24599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:18.416411 24599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:18.416515 24599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:18.423768 24599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:18.423919 24599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:18.424052 24599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:40:18.434017 24599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:18.435284 24599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:18.436724 24599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:18.450052 24607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:18.450138 24604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:18.450093 24605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:18.451263 24599 server_base.cc:1034] running on GCE node
I20260926 16:40:18.451965 24599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:18.453022 24599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:18.454233 24599 hybrid_clock.cc:648] HybridClock initialized: now 1790440818454176 us; error 38 us; skew 500 ppm
I20260926 16:40:18.454547 24599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:18 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440818, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:18.457801 24599 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:40:18.459422 24599 webserver.cc:460] Webserver started at http://127.10.110.1:45317/ using document root <none> and password file <none>
I20260926 16:40:18.460191 24599 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:18.460304 24599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:18.460558 24599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:18.463418 24599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "412834bd8a434356ba5bcb75d66c0826"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "e543a4b9cbf26c339dc71c301285ef31"
server_key_iv: "c5cf5e6156e0d1e36d556c8e6b20e9c1"
server_key_version: "encryptionkey@0"
I20260926 16:40:18.464202 24599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "412834bd8a434356ba5bcb75d66c0826"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "e543a4b9cbf26c339dc71c301285ef31"
server_key_iv: "c5cf5e6156e0d1e36d556c8e6b20e9c1"
server_key_version: "encryptionkey@0"
I20260926 16:40:18.469511 24599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:40:18.473070 24614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:18.474475 24599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:18.474630 24599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "412834bd8a434356ba5bcb75d66c0826"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "e543a4b9cbf26c339dc71c301285ef31"
server_key_iv: "c5cf5e6156e0d1e36d556c8e6b20e9c1"
server_key_version: "encryptionkey@0"
I20260926 16:40:18.474828 24599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:18.491163 24599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:18.495335 24599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:18.495611 24599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:18.496622 24599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:18.498052 24599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:18.498129 24599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:18.498260 24599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:18.498315 24599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:18.528847 24599 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:33019
I20260926 16:40:18.528891 24726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:33019 every 8 connection(s)
I20260926 16:40:18.530104 24599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:40:18.538062 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24599
I20260926 16:40:18.538254 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:40:18.538897 10680 external_mini_cluster.cc:1428] Setting key cf698e93e1d84619b7ed361a38afc51b
Sep 26 16:40:18 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440818, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:18.550323 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:18.532698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:36869 (local address 127.10.110.62:36663)
0926 16:40:18.533091 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:18.533095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:18.533946 (+ 851us) server_negotiation.cc:407] Connection header received
0926 16:40:18.535704 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:18.535710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:18.535793 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:18.535920 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:18.537448 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:18.538145 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:18.539822 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:18.540425 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:18.540549 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:18.544112 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:18.544118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:18.544123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:18.544166 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:18.547075 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:18.547772 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:18.547778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:18.547781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:18.547955 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:18.548479 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:18.548482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:18.548484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:18.548761 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:18.548868 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:18.549872 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0926 16:40:18.550055 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":190,"threads_started":1}
I20260926 16:40:18.551816 24730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:18.533085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:36869)
0926 16:40:18.533766 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:18.533843 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:18.535439 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:18.536155 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:18.536163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:18.536684 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:18.537278 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:18.537295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:18.538344 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:18.538350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:18.539658 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:18.539673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:18.540636 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:18.540644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:18.540881 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:18.541535 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:18.541558 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:18.543862 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:18.547224 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:18.547231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:18.547246 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:18.547662 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:18.548056 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:18.548059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:18.548061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:18.548264 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:18.548907 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:18.548917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:18.549423 (+ 506us) client_negotiation.cc:769] Sending connection context
0926 16:40:18.549769 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 16:40:18.550366 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260926 16:40:18.553411 24727 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36663
I20260926 16:40:18.553766 24727 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:18.554756 24727 heartbeater.cc:511] Master 127.10.110.62:36663 requested a full tablet report, sending...
I20260926 16:40:18.557376 24538 ts_manager.cc:194] Registered new tserver with Master: 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1:33019)
I20260926 16:40:18.559494 24538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:36869
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:40:18.594452 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36663
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:18.933736 24733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:18.934024 24733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:18.934140 24733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:18.941488 24733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:18.941622 24733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:18.941773 24733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:40:18.951704 24733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:18.953073 24733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:18.954456 24733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:18.967870 24738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:18.967788 24741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:18.967813 24739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:18.968634 24733 server_base.cc:1034] running on GCE node
I20260926 16:40:18.969256 24733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:18.970218 24733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:18.971449 24733 hybrid_clock.cc:648] HybridClock initialized: now 1790440818971391 us; error 44 us; skew 500 ppm
I20260926 16:40:18.971856 24733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:18 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440818, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:18.975183 24733 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:40:18.976886 24733 webserver.cc:460] Webserver started at http://127.10.110.2:42573/ using document root <none> and password file <none>
I20260926 16:40:18.977607 24733 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:18.977720 24733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:18.978018 24733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:18.981591 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "4e48311d4b0d4d8793b09d5c020908ff"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "ceeb4e3bcb7e8dc3489a1d158ddec0b0"
server_key_iv: "4ed932c1f6d2e16f501aa7bb09b13f2f"
server_key_version: "encryptionkey@0"
I20260926 16:40:18.982393 24733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "4e48311d4b0d4d8793b09d5c020908ff"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "ceeb4e3bcb7e8dc3489a1d158ddec0b0"
server_key_iv: "4ed932c1f6d2e16f501aa7bb09b13f2f"
server_key_version: "encryptionkey@0"
I20260926 16:40:18.987542 24733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:40:18.991230 24748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:18.992763 24733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:18.992928 24733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "4e48311d4b0d4d8793b09d5c020908ff"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "ceeb4e3bcb7e8dc3489a1d158ddec0b0"
server_key_iv: "4ed932c1f6d2e16f501aa7bb09b13f2f"
server_key_version: "encryptionkey@0"
I20260926 16:40:18.993152 24733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:19.015380 24733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:19.019776 24733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:19.020083 24733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:19.021065 24733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:19.022442 24733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:19.022519 24733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:19.022646 24733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:19.022706 24733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:19.055337 24733 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:38445
I20260926 16:40:19.055403 24860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:38445 every 8 connection(s)
I20260926 16:40:19.056787 24733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:40:19.066700 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24733
I20260926 16:40:19.066887 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:40:19.067358 10680 external_mini_cluster.cc:1428] Setting key e4c16411e154a7e962b0373fa7f4ea9a
Sep 26 16:40:19 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440818, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:19.077548 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.059153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:49215 (local address 127.10.110.62:36663)
0926 16:40:19.059570 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.059575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.060435 (+ 860us) server_negotiation.cc:407] Connection header received
0926 16:40:19.062279 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.062286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.062365 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.062460 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:19.063972 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.064692 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.066238 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.066663 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.066771 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.070767 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:19.070775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:19.070780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:19.070821 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:19.074132 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.074961 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.074967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.074971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.075124 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.075665 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.075669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.075671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.076000 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:19.076126 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.077061 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.077253 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":202,"threads_started":1}
I20260926 16:40:19.079011 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.059546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:49215)
0926 16:40:19.060273 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.060320 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.062010 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.062704 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.062720 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.063231 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:19.063783 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.063802 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.064853 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.064857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.066091 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.066104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.066883 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.066890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.067131 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.067868 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:19.067905 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:19.070513 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:19.074304 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:19.074312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:19.074328 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:19.074831 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:19.075267 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:19.075272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:19.075275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:19.075531 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:19.076149 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:19.076157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:19.076620 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.076916 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.077519 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":295,"threads_started":1}
I20260926 16:40:19.080482 24861 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36663
I20260926 16:40:19.080813 24861 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:19.081746 24861 heartbeater.cc:511] Master 127.10.110.62:36663 requested a full tablet report, sending...
I20260926 16:40:19.083310 24538 ts_manager.cc:194] Registered new tserver with Master: 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445)
I20260926 16:40:19.084210 24538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:49215
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:40:19.120060 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36663
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:19.455727 24867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:19.456086 24867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:19.456197 24867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:19.463619 24867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:19.463752 24867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:19.463910 24867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:40:19.474038 24867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:19.475342 24867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:19.476755 24867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:19.489799 24875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:19.489845 24873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:19.489763 24872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:19.490892 24867 server_base.cc:1034] running on GCE node
I20260926 16:40:19.491551 24867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:19.492722 24867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:19.493929 24867 hybrid_clock.cc:648] HybridClock initialized: now 1790440819493853 us; error 68 us; skew 500 ppm
I20260926 16:40:19.494271 24867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:19 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440819, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:19.497539 24867 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:40:19.499130 24867 webserver.cc:460] Webserver started at http://127.10.110.3:37223/ using document root <none> and password file <none>
I20260926 16:40:19.499869 24867 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:19.500000 24867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:19.500295 24867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:19.503332 24867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "aac387ee061246a396cf23f246551091"
format_stamp: "Formatted at 2026-09-26 16:40:19 on dist-test-slave-whjp"
server_key: "2c3f53284c0a686ea5cccc97cb5c76f5"
server_key_iv: "15da77f441b82cf113ee036fa3ee8096"
server_key_version: "encryptionkey@0"
I20260926 16:40:19.504184 24867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "aac387ee061246a396cf23f246551091"
format_stamp: "Formatted at 2026-09-26 16:40:19 on dist-test-slave-whjp"
server_key: "2c3f53284c0a686ea5cccc97cb5c76f5"
server_key_iv: "15da77f441b82cf113ee036fa3ee8096"
server_key_version: "encryptionkey@0"
I20260926 16:40:19.509450 24867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:40:19.513145 24882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:19.514609 24867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:19.514771 24867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "aac387ee061246a396cf23f246551091"
format_stamp: "Formatted at 2026-09-26 16:40:19 on dist-test-slave-whjp"
server_key: "2c3f53284c0a686ea5cccc97cb5c76f5"
server_key_iv: "15da77f441b82cf113ee036fa3ee8096"
server_key_version: "encryptionkey@0"
I20260926 16:40:19.514969 24867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:19.542546 24867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:19.547016 24867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:19.547295 24867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:19.548287 24867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:19.549636 24867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:19.549712 24867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:19.549832 24867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:19.549882 24867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:19.563575 24727 heartbeater.cc:503] Master 127.10.110.62:36663 was elected leader, sending a full tablet report...
I20260926 16:40:19.580942 24867 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38651
I20260926 16:40:19.580991 24994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38651 every 8 connection(s)
I20260926 16:40:19.582235 24867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:40:19.592600 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 24867
I20260926 16:40:19.592818 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:40:19.593389 10680 external_mini_cluster.cc:1428] Setting key 06157902662042448fe6e6bde1765cdf
Sep 26 16:40:19 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440819, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:19.603083 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.584762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:37263 (local address 127.10.110.62:36663)
0926 16:40:19.585243 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.585249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.586110 (+ 861us) server_negotiation.cc:407] Connection header received
0926 16:40:19.588032 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.588037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.588123 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.588218 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:19.590010 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.590860 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.592717 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.593342 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.593501 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.597395 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:19.597402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:19.597407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:19.597444 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:19.600072 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.600854 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.600858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.600861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.601027 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.601456 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.601461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.601463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.601755 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:19.601877 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.602628 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.602806 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":208,"threads_started":1}
I20260926 16:40:19.604156 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.585239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:37263)
0926 16:40:19.585951 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.585995 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.587701 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.588424 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.588433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.589036 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:19.589797 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.589824 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.591093 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.591099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.592524 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.592540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.593638 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.593646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.593864 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.594658 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:19.594690 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:19.597157 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:19.600236 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:19.600244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:19.600258 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:19.600737 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:19.601109 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:19.601112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:19.601115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:19.601327 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:19.601858 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:19.601864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:19.602203 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.602491 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.602936 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":284,"threads_started":1}
I20260926 16:40:19.605584 24995 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36663
I20260926 16:40:19.605876 24995 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:19.606608 24995 heartbeater.cc:511] Master 127.10.110.62:36663 requested a full tablet report, sending...
I20260926 16:40:19.608192 24538 ts_manager.cc:194] Registered new tserver with Master: aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
I20260926 16:40:19.609159 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:40:19.609184 24538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:37263
I20260926 16:40:19.625521 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.613615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.10.110.62:36663)
0926 16:40:19.613841 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.613849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.614190 (+ 341us) server_negotiation.cc:407] Connection header received
0926 16:40:19.614334 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.614338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.614400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.614477 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:19.615914 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.616701 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.617987 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.618404 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.618494 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.620803 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:19.620808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:19.620812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:19.620847 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:19.623273 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.623918 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.623921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.623924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.624043 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.624345 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.624348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.624350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.624665 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:19.624853 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.625106 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.625288 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:40:19.629886 24538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:40:19.637306 24538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:40:19.670290 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.650530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.10.110.1:33019)
0926 16:40:19.652117 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.652129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.652162 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:40:19.652291 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.652300 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.652608 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.652830 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:19.660235 (+ 7405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.662966 (+ 2731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.666369 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.668121 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.668267 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.668533 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.669668 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.669872 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":1204,"threads_started":1}
I20260926 16:40:19.671119 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.650305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58154)
0926 16:40:19.651122 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.651182 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.651382 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.653169 (+ 1787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.653179 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.653230 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:19.659878 (+ 6648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.659899 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.663272 (+ 3373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.663280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.666097 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.666124 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.668676 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.668683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.669354 (+ 671us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.669436 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.670175 (+ 739us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.670636 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":145,"thread_start_us":394,"threads_started":1}
I20260926 16:40:19.674479 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.651110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.10.110.2:38445)
0926 16:40:19.666776 (+ 15666us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.666787 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.666822 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:40:19.666943 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.666953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.667282 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.667532 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:19.668271 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.669748 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.672222 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.673347 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.673489 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.673723 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.673996 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.674172 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15426,"thread_start_us":16638,"threads_started":1}
I20260926 16:40:19.674530 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.650448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:38445 (local address 127.0.0.1:42092)
0926 16:40:19.663274 (+ 12826us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.663303 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.663466 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.667524 (+ 4058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.667529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.667563 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:19.667960 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.667976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.669914 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.669918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.672030 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.672049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.673603 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.673614 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.673795 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.673828 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.673906 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.674053 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12665,"mutex_wait_us":1,"thread_start_us":21064,"threads_started":1}
I20260926 16:40:19.675776 24662 tablet_service.cc:1467] Processing CreateTablet for tablet b01c60d170c84858815129122113eb64 (DEFAULT_TABLE table=test-table [id=c1ff6e6665b6451ea3b0437a942ab213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:19.677639 24662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01c60d170c84858815129122113eb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:19.678432 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.652999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38651 (local address 127.0.0.1:33398)
0926 16:40:19.668443 (+ 15444us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.668470 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.668598 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.669888 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.669893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.669920 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:19.670439 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.670453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.672571 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.672577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.674866 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.674885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.677380 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.677386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.677656 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.677691 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.677786 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.677936 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15299,"thread_start_us":9929,"threads_started":1}
I20260926 16:40:19.678869 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.653663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.10.110.3:38651)
0926 16:40:19.669231 (+ 15568us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.669240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.669274 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:40:19.669375 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.669382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.669647 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.670556 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:19.670870 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.672358 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.675083 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.676991 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.677332 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.677585 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.677938 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.678313 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15413,"thread_start_us":15255,"threads_started":1}
I20260926 16:40:19.679008 24796 tablet_service.cc:1467] Processing CreateTablet for tablet b01c60d170c84858815129122113eb64 (DEFAULT_TABLE table=test-table [id=c1ff6e6665b6451ea3b0437a942ab213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:19.683256 24796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01c60d170c84858815129122113eb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:19.683293 24930 tablet_service.cc:1467] Processing CreateTablet for tablet b01c60d170c84858815129122113eb64 (DEFAULT_TABLE table=test-table [id=c1ff6e6665b6451ea3b0437a942ab213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:19.685106 24930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01c60d170c84858815129122113eb64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:19.693763 25014 tablet_bootstrap.cc:492] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Bootstrap starting.
I20260926 16:40:19.696614 25015 tablet_bootstrap.cc:492] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Bootstrap starting.
I20260926 16:40:19.697584 25014 tablet_bootstrap.cc:654] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:19.698742 25014 log.cc:824] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:19.700352 25017 tablet_bootstrap.cc:492] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Bootstrap starting.
I20260926 16:40:19.700788 25015 tablet_bootstrap.cc:654] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:19.701988 25015 log.cc:824] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:19.702461 25014 tablet_bootstrap.cc:492] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: No bootstrap required, opened a new log
I20260926 16:40:19.702780 25014 ts_tablet_manager.cc:1397] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:40:19.704787 25017 tablet_bootstrap.cc:654] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:19.706220 25017 log.cc:824] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:19.709824 25015 tablet_bootstrap.cc:492] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: No bootstrap required, opened a new log
I20260926 16:40:19.709824 25017 tablet_bootstrap.cc:492] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: No bootstrap required, opened a new log
I20260926 16:40:19.710145 25017 ts_tablet_manager.cc:1397] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 16:40:19.710145 25015 ts_tablet_manager.cc:1397] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260926 16:40:19.711938 25014 raft_consensus.cc:348] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.712376 25014 raft_consensus.cc:374] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:19.712527 25014 raft_consensus.cc:729] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412834bd8a434356ba5bcb75d66c0826, State: Initialized, Role: FOLLOWER
I20260926 16:40:19.713186 25014 consensus_queue.cc:260] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.715241 25014 ts_tablet_manager.cc:1428] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 16:40:19.721164 25017 raft_consensus.cc:348] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.721164 25015 raft_consensus.cc:348] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.721659 25015 raft_consensus.cc:374] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:19.721659 25017 raft_consensus.cc:374] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:19.721767 25017 raft_consensus.cc:729] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac387ee061246a396cf23f246551091, State: Initialized, Role: FOLLOWER
I20260926 16:40:19.721767 25015 raft_consensus.cc:729] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e48311d4b0d4d8793b09d5c020908ff, State: Initialized, Role: FOLLOWER
I20260926 16:40:19.722445 25017 consensus_queue.cc:260] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.722445 25015 consensus_queue.cc:260] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.723595 24861 heartbeater.cc:503] Master 127.10.110.62:36663 was elected leader, sending a full tablet report...
I20260926 16:40:19.724102 24995 heartbeater.cc:503] Master 127.10.110.62:36663 was elected leader, sending a full tablet report...
I20260926 16:40:19.724519 25015 ts_tablet_manager.cc:1428] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 16:40:19.724608 25017 ts_tablet_manager.cc:1428] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260926 16:40:19.731796 25021 raft_consensus.cc:482] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:19.732132 25021 raft_consensus.cc:504] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.733891 25021 leader_election.cc:288] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1:33019), 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445)
I20260926 16:40:19.740440 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.734158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.10.110.3:40277)
0926 16:40:19.734351 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.734371 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.734500 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.735404 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.735408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.735432 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:19.735686 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.735700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.736853 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.736859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.738415 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.738431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.739703 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.739710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.739933 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.739970 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.740072 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.740193 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":57}
I20260926 16:40:19.740526 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.734840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:40277 (local address 127.10.110.1:33019)
0926 16:40:19.735035 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.735040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.735059 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:40:19.735134 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.735139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.735232 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.735312 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:19.735856 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.736714 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.738595 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.739494 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.739624 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.739683 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.740120 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.740281 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:40:19.740787 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.734496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:54589 (local address 127.10.110.2:38445)
0926 16:40:19.734698 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.734703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.735297 (+ 594us) server_negotiation.cc:407] Connection header received
0926 16:40:19.735553 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.735558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.735635 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.735735 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:19.736277 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.737034 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.738897 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.739766 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.739972 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.740043 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.740410 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.740545 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:40:19.741226 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.734179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:38445 (local address 127.10.110.3:54589)
0926 16:40:19.735200 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:19.735217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:19.735395 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:19.735769 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:19.735773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:19.735799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:19.736138 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.736151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.737186 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.737189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.738723 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:19.738738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.740041 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.740048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:19.740240 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.740272 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:40:19.740377 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 16:40:19.740539 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":151,"thread_start_us":513,"threads_started":1}
I20260926 16:40:19.741833 24682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01c60d170c84858815129122113eb64" candidate_uuid: "aac387ee061246a396cf23f246551091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412834bd8a434356ba5bcb75d66c0826" is_pre_election: true
I20260926 16:40:19.742241 24816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01c60d170c84858815129122113eb64" candidate_uuid: "aac387ee061246a396cf23f246551091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e48311d4b0d4d8793b09d5c020908ff" is_pre_election: true
I20260926 16:40:19.742549 24682 raft_consensus.cc:2454] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac387ee061246a396cf23f246551091 in term 0.
I20260926 16:40:19.742789 24816 raft_consensus.cc:2454] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac387ee061246a396cf23f246551091 in term 0.
I20260926 16:40:19.743511 24883 leader_election.cc:302] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826, aac387ee061246a396cf23f246551091; no voters:
I20260926 16:40:19.743939 25021 raft_consensus.cc:2781] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:40:19.744086 25021 raft_consensus.cc:482] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:40:19.744190 25021 raft_consensus.cc:3037] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:19.746367 25021 raft_consensus.cc:504] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.747064 25021 leader_election.cc:288] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [CANDIDATE]: Term 1 election: Requested vote from peers 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1:33019), 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445)
I20260926 16:40:19.747553 24682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01c60d170c84858815129122113eb64" candidate_uuid: "aac387ee061246a396cf23f246551091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412834bd8a434356ba5bcb75d66c0826"
I20260926 16:40:19.747632 24816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01c60d170c84858815129122113eb64" candidate_uuid: "aac387ee061246a396cf23f246551091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e48311d4b0d4d8793b09d5c020908ff"
I20260926 16:40:19.747794 24682 raft_consensus.cc:3037] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:19.747860 24816 raft_consensus.cc:3037] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:19.751008 24816 raft_consensus.cc:2454] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac387ee061246a396cf23f246551091 in term 1.
I20260926 16:40:19.751008 24682 raft_consensus.cc:2454] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac387ee061246a396cf23f246551091 in term 1.
I20260926 16:40:19.751534 24883 leader_election.cc:302] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826, 4e48311d4b0d4d8793b09d5c020908ff, aac387ee061246a396cf23f246551091; no voters:
I20260926 16:40:19.751945 25021 raft_consensus.cc:2781] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:19.752406 25021 raft_consensus.cc:686] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 1 LEADER]: Becoming Leader. State: Replica: aac387ee061246a396cf23f246551091, State: Running, Role: LEADER
I20260926 16:40:19.752946 25021 consensus_queue.cc:237] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:19.757859 24538 catalog_manager.cc:5495] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 reported cstate change: term changed from 0 to 1, leader changed from <none> to aac387ee061246a396cf23f246551091 (127.10.110.3). New cstate: current_term: 1 leader_uuid: "aac387ee061246a396cf23f246551091" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } health_report { overall_health: HEALTHY } } }
W20260926 16:40:19.782291 24728 tablet.cc:2369] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:19.792291 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.786713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.10.110.3:38651)
0926 16:40:19.786956 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.786962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.786985 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:40:19.787159 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.787163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.787249 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.787395 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:40:19.787854 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.788595 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.790021 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.790442 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.790557 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.791139 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:40:19.791657 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:40:19.791803 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.791940 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.792028 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:40:19.800999 24682 raft_consensus.cc:1261] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Refusing update from remote peer aac387ee061246a396cf23f246551091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:19.800998 24816 raft_consensus.cc:1261] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Refusing update from remote peer aac387ee061246a396cf23f246551091: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:19.802079 25024 consensus_queue.cc:1035] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:19.802421 25021 consensus_queue.cc:1035] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:40:19.808672 24862 tablet.cc:2369] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:19.814199 25027 mvcc.cc:205] Tried to move back new op lower bound from 7333645597894742016 to 7333645597712338944. Current Snapshot: MvccSnapshot[applied={T|T < 7333645597894742016}]
I20260926 16:40:19.830761 24537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:19.831121 24537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:19.835775 24537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Sending DeleteTablet for 3 replicas of tablet b01c60d170c84858815129122113eb64
I20260926 16:40:19.837152 24662 tablet_service.cc:1514] Processing DeleteTablet for tablet b01c60d170c84858815129122113eb64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:19 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:58154
I20260926 16:40:19.837163 24796 tablet_service.cc:1514] Processing DeleteTablet for tablet b01c60d170c84858815129122113eb64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:19 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42092
I20260926 16:40:19.837338 24930 tablet_service.cc:1514] Processing DeleteTablet for tablet b01c60d170c84858815129122113eb64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:19 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:33398
I20260926 16:40:19.838665 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24599
I20260926 16:40:19.865795 25038 tablet_replica.cc:330] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: stopping tablet replica
I20260926 16:40:19.866421 25038 raft_consensus.cc:2229] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:19.867087 25038 raft_consensus.cc:2258] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:19.873148 25038 ts_tablet_manager.cc:1905] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:19.880396 25039 tablet_replica.cc:330] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: stopping tablet replica
I20260926 16:40:19.880843 25037 tablet_replica.cc:330] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: stopping tablet replica
I20260926 16:40:19.881422 25037 raft_consensus.cc:2229] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:19.882001 25039 raft_consensus.cc:2229] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 1 LEADER]: Raft consensus shutting down.
Sep 26 16:40:19 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:40:19.882624 25037 raft_consensus.cc:2258] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:19.883364 25039 raft_consensus.cc:2258] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:19.884188 25038 ts_tablet_manager.cc:1918] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:19.886399 25037 ts_tablet_manager.cc:1905] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:19.886605 25039 ts_tablet_manager.cc:1905] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:19.890028 25038 log.cc:1196] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/b01c60d170c84858815129122113eb64
I20260926 16:40:19.890966 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:19.840026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.10.110.1:33019)
0926 16:40:19.840232 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:19.840239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:19.862526 (+ 22287us) server_negotiation.cc:407] Connection header received
0926 16:40:19.863106 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:19.863113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:19.863224 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:19.863307 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:19.867340 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.868551 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.873441 (+ 4890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:19.874073 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:19.874195 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:19.883458 (+ 9263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:19.883473 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:19.883492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:19.883542 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:19.887122 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.888073 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.888083 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.888086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.888258 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:19.888747 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:19.888752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:19.888755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:19.889207 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:19.889504 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:19.890357 (+ 853us) server_negotiation.cc:299] Negotiation successful
0926 16:40:19.890587 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:40:19.892431 25038 ts_tablet_manager.cc:1939] T b01c60d170c84858815129122113eb64 P 412834bd8a434356ba5bcb75d66c0826: Deleting consensus metadata
I20260926 16:40:19.892663 24722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:19.893934 25039 ts_tablet_manager.cc:1918] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:19.894163 25039 log.cc:1196] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/b01c60d170c84858815129122113eb64
I20260926 16:40:19.894773 25039 ts_tablet_manager.cc:1939] T b01c60d170c84858815129122113eb64 P aac387ee061246a396cf23f246551091: Deleting consensus metadata
I20260926 16:40:19.894963 25037 ts_tablet_manager.cc:1918] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:19.895179 25037 log.cc:1196] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/b01c60d170c84858815129122113eb64
I20260926 16:40:19.895704 25037 ts_tablet_manager.cc:1939] T b01c60d170c84858815129122113eb64 P 4e48311d4b0d4d8793b09d5c020908ff: Deleting consensus metadata
I20260926 16:40:19.896680 24523 catalog_manager.cc:4841] TS aac387ee061246a396cf23f246551091 (127.10.110.3:38651): tablet b01c60d170c84858815129122113eb64 (table test-table [id=c1ff6e6665b6451ea3b0437a942ab213]) successfully deleted
I20260926 16:40:19.898074 24525 catalog_manager.cc:4841] TS 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): tablet b01c60d170c84858815129122113eb64 (table test-table [id=c1ff6e6665b6451ea3b0437a942ab213]) successfully deleted
I20260926 16:40:20.034204 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24599
I20260926 16:40:20.034262 24522 catalog_manager.cc:4841] TS 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1:33019): tablet b01c60d170c84858815129122113eb64 (table test-table [id=c1ff6e6665b6451ea3b0437a942ab213]) successfully deleted
I20260926 16:40:20.048754 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24733
Sep 26 16:40:20 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:40:20.068761 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:20.049603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.10.110.2:38445)
0926 16:40:20.049792 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:20.049798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:20.049821 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:40:20.049992 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:20.049997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:20.050079 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:20.050161 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:20.052023 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:20.052771 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:20.054861 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:20.055506 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:20.055635 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:20.061272 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:20.061288 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:20.061311 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:20.061370 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:20.065341 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:20.066323 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:20.066333 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:20.066337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:20.066495 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:20.066877 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:20.066882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:20.066885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:20.067424 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:20.067719 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:20.068135 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 16:40:20.068375 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:40:20.069576 24856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:20.160693 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24733
I20260926 16:40:20.181579 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24867
Sep 26 16:40:20 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:40:20.200492 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:20.182420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.10.110.3:38651)
0926 16:40:20.182570 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:20.182574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:20.182593 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:40:20.182748 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:20.182752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:20.182815 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:20.182897 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:20.184731 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:20.185576 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:20.187450 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:20.187960 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:20.188055 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:20.193774 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:20.193792 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:20.193809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:20.193859 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:20.196941 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:20.197936 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:20.197944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:20.197946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:20.198097 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:20.198580 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:20.198584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:20.198586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:20.198984 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:20.199225 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:20.199681 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:40:20.200095 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:40:20.201282 24990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:20.328112 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24867
I20260926 16:40:20.344903 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 24506
I20260926 16:40:20.346024 24568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:20.460070 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 24506
I20260926 16:40:20.477900 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36663
--webserver_interface=127.10.110.62
--webserver_port=34485
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:20.804469 25045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:20.804777 25045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:20.804883 25045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:20.812438 25045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:20.812552 25045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:20.812606 25045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:20.812664 25045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:20.812713 25045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:20.822670 25045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=34485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:20.823942 25045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:20.825193 25045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:20.846089 25051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:20.845995 25052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:20.846482 25054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:20.847754 25045 server_base.cc:1034] running on GCE node
I20260926 16:40:20.848727 25045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:20.850407 25045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:20.851604 25045 hybrid_clock.cc:648] HybridClock initialized: now 1790440820851556 us; error 77 us; skew 500 ppm
I20260926 16:40:20.852062 25045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:20 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440820, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:20.855419 25045 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:20.857084 25045 webserver.cc:460] Webserver started at http://127.10.110.62:34485/ using document root <none> and password file <none>
I20260926 16:40:20.857867 25045 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:20.858002 25045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:20.863873 25045 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:40:20.866879 25061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:20.868319 25045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:20.868484 25045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "1731653a8faf450691ff92da350052d7"
format_stamp: "Formatted at 2026-09-26 16:40:17 on dist-test-slave-whjp"
server_key: "af32c31a5e10a1eff892944e0c403cd7"
server_key_iv: "252746bc97ee7c8e5d134d36b4fe509b"
server_key_version: "encryptionkey@0"
I20260926 16:40:20.869599 25045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:20.900256 25045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:20.905036 25045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:20.905335 25045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:20.924140 25045 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:36663
I20260926 16:40:20.924121 25112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:36663 every 8 connection(s)
I20260926 16:40:20.925647 25045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:20.930279 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25045
I20260926 16:40:20.931231 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:33019
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:36663
--webserver_port=45317
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:20.933864 25113 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:40:20.938659 25113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap starting.
I20260926 16:40:20.959045 25113 log.cc:824] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:20.971935 25113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:40:20.972493 25113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap complete.
I20260926 16:40:20.982244 25113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:20.983232 25113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1731653a8faf450691ff92da350052d7, State: Initialized, Role: FOLLOWER
I20260926 16:40:20.983942 25113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:20.984196 25113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:20.984325 25113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:20.984496 25113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:40:20.988855 25113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:20.989415 25113 leader_election.cc:302] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1731653a8faf450691ff92da350052d7; no voters:
I20260926 16:40:20.990031 25113 leader_election.cc:288] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:40:20.990181 25117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:40:20.991250 25117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 LEADER]: Becoming Leader. State: Replica: 1731653a8faf450691ff92da350052d7, State: Running, Role: LEADER
I20260926 16:40:20.991922 25117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:20.992393 25113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:20.994334 25119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1731653a8faf450691ff92da350052d7. Latest consensus state: current_term: 2 leader_uuid: "1731653a8faf450691ff92da350052d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } } }
I20260926 16:40:20.994764 25118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1731653a8faf450691ff92da350052d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } } }
I20260926 16:40:20.995518 25118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:20.995853 25119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:21.012991 25124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:21.021108 25124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:21.022720 25124 catalog_manager.cc:1259] Loaded cluster ID: 7a0f832428924585b7ddbd1c02f19757
I20260926 16:40:21.022910 25124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:21.026275 25124 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 16:40:21.040299 25124 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 16:40:21.047122 25124 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:21.048616 25124 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Loaded TSK: 0
I20260926 16:40:21.049836 25124 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:40:21.290952 25115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:21.291298 25115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:21.291433 25115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:21.298998 25115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:21.299124 25115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:21.299276 25115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:40:21.309299 25115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:33019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=45317
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:21.310667 25115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:21.312021 25115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:21.325171 25140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:21.325146 25141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:21.326607 25143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:21.326680 25115 server_base.cc:1034] running on GCE node
I20260926 16:40:21.327327 25115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:21.328354 25115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:21.329569 25115 hybrid_clock.cc:648] HybridClock initialized: now 1790440821329525 us; error 37 us; skew 500 ppm
I20260926 16:40:21.329876 25115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:21 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440821, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:21.338402 25115 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:40:21.339967 25115 webserver.cc:460] Webserver started at http://127.10.110.1:45317/ using document root <none> and password file <none>
I20260926 16:40:21.340649 25115 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:21.340772 25115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:21.346204 25115 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:40:21.348933 25150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:21.350381 25115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:21.350533 25115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "412834bd8a434356ba5bcb75d66c0826"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "e543a4b9cbf26c339dc71c301285ef31"
server_key_iv: "c5cf5e6156e0d1e36d556c8e6b20e9c1"
server_key_version: "encryptionkey@0"
I20260926 16:40:21.351517 25115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:21.365126 25115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:21.369385 25115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:21.369640 25115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:21.370697 25115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:21.372135 25115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:21.372224 25115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:21.372323 25115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:21.372396 25115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:21.402420 25115 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:33019
I20260926 16:40:21.402487 25262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:33019 every 8 connection(s)
I20260926 16:40:21.403805 25115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:40:21.406104 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25115
I20260926 16:40:21.406853 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:38445
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:36663
--webserver_port=42573
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:40:21 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440821, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:21.429858 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:21.406544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:59595 (local address 127.10.110.62:36663)
0926 16:40:21.407200 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:21.407217 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:21.410410 (+ 3193us) server_negotiation.cc:407] Connection header received
0926 16:40:21.410769 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:21.410787 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:21.411239 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:21.411600 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:21.413885 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.415698 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.417222 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.418137 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.418313 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:21.421798 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:21.421807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:21.421822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:21.421867 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:21.424997 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:21.426031 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:21.426038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:21.426044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:21.426237 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:21.426828 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:21.426832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:21.426835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:21.427336 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:21.427499 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:21.428506 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 16:40:21.428890 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":250,"threads_started":1}
I20260926 16:40:21.430197 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:21.406633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:59595)
0926 16:40:21.407731 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:21.407777 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:21.409999 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:21.412308 (+ 2309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:21.412318 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:21.412947 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:21.413636 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:21.413659 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.415884 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.415888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:21.417079 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:21.417095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.418345 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.418350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:21.418546 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:21.419153 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:21.419175 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:21.421451 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:21.425173 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:21.425183 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:21.425199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:21.425841 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:21.426363 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:21.426368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:21.426372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:21.426634 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:21.427482 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:21.427491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:21.427977 (+ 486us) client_negotiation.cc:769] Sending connection context
0926 16:40:21.428314 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 16:40:21.428925 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":615,"threads_started":1}
I20260926 16:40:21.433333 25263 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36663
I20260926 16:40:21.433607 25263 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:21.434371 25263 heartbeater.cc:511] Master 127.10.110.62:36663 requested a full tablet report, sending...
I20260926 16:40:21.436653 25078 ts_manager.cc:194] Registered new tserver with Master: 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1:33019)
I20260926 16:40:21.438773 25078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:59595
W20260926 16:40:21.741614 25267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:21.741885 25267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:21.741976 25267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:21.749199 25267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:21.749320 25267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:21.749465 25267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:40:21.759533 25267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:38445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=42573
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:21.761081 25267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:21.762373 25267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:21.774549 25272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:21.774761 25273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:21.774799 25275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:21.775723 25267 server_base.cc:1034] running on GCE node
I20260926 16:40:21.776342 25267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:21.777259 25267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:21.778462 25267 hybrid_clock.cc:648] HybridClock initialized: now 1790440821778410 us; error 46 us; skew 500 ppm
I20260926 16:40:21.778756 25267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:21 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440821, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:21.781723 25267 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:40:21.783177 25267 webserver.cc:460] Webserver started at http://127.10.110.2:42573/ using document root <none> and password file <none>
I20260926 16:40:21.783989 25267 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:21.784142 25267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:21.789486 25267 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:21.792256 25282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:21.793645 25267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:40:21.793804 25267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "4e48311d4b0d4d8793b09d5c020908ff"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "ceeb4e3bcb7e8dc3489a1d158ddec0b0"
server_key_iv: "4ed932c1f6d2e16f501aa7bb09b13f2f"
server_key_version: "encryptionkey@0"
I20260926 16:40:21.794764 25267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:21.820770 25267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:21.825235 25267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:21.825489 25267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:21.826493 25267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:21.827777 25267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:21.827899 25267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:21.827993 25267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:21.828048 25267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:21.856459 25267 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:38445
I20260926 16:40:21.856523 25394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:38445 every 8 connection(s)
I20260926 16:40:21.857821 25267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:40:21.861714 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25267
I20260926 16:40:21.862287 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:38651
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:36663
--webserver_port=37223
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:40:21 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440821, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:21.878461 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:21.860310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:39943 (local address 127.10.110.62:36663)
0926 16:40:21.860550 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:21.860554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:21.861703 (+ 1149us) server_negotiation.cc:407] Connection header received
0926 16:40:21.863618 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:21.863625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:21.863729 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:21.863847 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:21.865378 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.866057 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.867608 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.868075 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.868179 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:21.872246 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:21.872252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:21.872255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:21.872295 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:21.875145 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:21.876080 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:21.876084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:21.876087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:21.876227 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:21.876686 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:21.876689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:21.876692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:21.876960 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:21.877108 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:21.878019 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 16:40:21.878213 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:40:21.880033 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:21.860744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:39943)
0926 16:40:21.861499 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:21.861542 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:21.863342 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:21.864095 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:21.864104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:21.864636 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:21.865202 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:21.865222 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.866225 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.866229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:21.867437 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:21.867448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:21.868292 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:21.868298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:21.868598 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:21.869269 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:21.869297 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:21.871980 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:21.875304 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:21.875313 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:21.875332 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:21.875940 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:21.876334 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:21.876339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:21.876343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:21.876584 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:21.877089 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:21.877097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:21.877582 (+ 485us) client_negotiation.cc:769] Sending connection context
0926 16:40:21.877911 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 16:40:21.878503 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":317,"threads_started":1}
I20260926 16:40:21.881505 25395 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36663
I20260926 16:40:21.881855 25395 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:21.882786 25395 heartbeater.cc:511] Master 127.10.110.62:36663 requested a full tablet report, sending...
I20260926 16:40:21.884438 25078 ts_manager.cc:194] Registered new tserver with Master: 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445)
I20260926 16:40:21.885260 25078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:39943
W20260926 16:40:22.186882 25398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:22.187170 25398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:22.187300 25398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:22.194792 25398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:22.194911 25398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:22.195060 25398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:40:22.205190 25398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:38651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=37223
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:22.206516 25398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:22.207811 25398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:22.220928 25404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:22.220989 25406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:22.221009 25403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:22.222059 25398 server_base.cc:1034] running on GCE node
I20260926 16:40:22.222662 25398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:22.223641 25398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:22.224879 25398 hybrid_clock.cc:648] HybridClock initialized: now 1790440822224806 us; error 58 us; skew 500 ppm
I20260926 16:40:22.225219 25398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:22 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440822, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:22.228457 25398 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:40:22.229918 25398 webserver.cc:460] Webserver started at http://127.10.110.3:37223/ using document root <none> and password file <none>
I20260926 16:40:22.230630 25398 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:22.230774 25398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:22.236268 25398 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:40:22.239154 25413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:22.240692 25398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:22.240865 25398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "aac387ee061246a396cf23f246551091"
format_stamp: "Formatted at 2026-09-26 16:40:19 on dist-test-slave-whjp"
server_key: "2c3f53284c0a686ea5cccc97cb5c76f5"
server_key_iv: "15da77f441b82cf113ee036fa3ee8096"
server_key_version: "encryptionkey@0"
I20260926 16:40:22.241878 25398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:22.268438 25398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:22.272660 25398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:22.272912 25398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:22.273950 25398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:22.275310 25398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:22.275393 25398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:22.275475 25398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:22.275530 25398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:22.305250 25398 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38651
I20260926 16:40:22.305328 25525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38651 every 8 connection(s)
I20260926 16:40:22.306658 25398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:40:22.315446 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25398
Sep 26 16:40:22 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440822, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:22.327316 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.309309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:53359 (local address 127.10.110.62:36663)
0926 16:40:22.309503 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.309510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.310643 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 16:40:22.312630 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.312636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.312721 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.312816 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:22.314321 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.315103 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.317208 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.317882 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.318025 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.321655 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:22.321660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:22.321663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:22.321696 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:22.324289 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.325005 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.325008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.325011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.325154 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.325582 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.325585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.325587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.325882 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:22.326037 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.326752 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.326984 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:40:22.328359 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.309706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:53359)
0926 16:40:22.310449 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.310530 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:22.312345 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:22.313062 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:22.313070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:22.313585 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:22.314142 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.314160 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.315293 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.315298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.317051 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.317077 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.318100 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.318105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.318299 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.318950 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:22.318970 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:22.321443 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:22.324419 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:22.324429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:22.324447 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:22.324865 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:22.325261 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:22.325267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:22.325271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:22.325458 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:22.326020 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:22.326028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:22.326380 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 16:40:22.326610 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 16:40:22.327152 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":273,"threads_started":1}
I20260926 16:40:22.329818 25526 heartbeater.cc:347] Connected to a master server at 127.10.110.62:36663
I20260926 16:40:22.330147 25526 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:22.330178 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.316380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.10.110.62:36663)
0926 16:40:22.316902 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.316908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.317254 (+ 346us) server_negotiation.cc:407] Connection header received
0926 16:40:22.317428 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.317431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.317491 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.317587 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:22.319356 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.320390 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.321748 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.322213 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.322323 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.324785 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:22.324789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:22.324793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:22.324822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:22.327444 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.328268 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.328272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.328274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.328383 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.328839 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.328844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.328846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.329119 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:22.329356 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.329634 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.329852 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
I20260926 16:40:22.330868 25526 heartbeater.cc:511] Master 127.10.110.62:36663 requested a full tablet report, sending...
I20260926 16:40:22.332284 25078 ts_manager.cc:194] Registered new tserver with Master: aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
I20260926 16:40:22.333132 25078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:53359
I20260926 16:40:22.333930 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:40:22.340407 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.334792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.10.110.62:36663)
0926 16:40:22.334970 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.334975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.335423 (+ 448us) server_negotiation.cc:407] Connection header received
0926 16:40:22.335583 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.335587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.335708 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.335819 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:40:22.336302 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.336993 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.338391 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.338804 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.338894 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.339399 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:40:22.339948 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:40:22.340057 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.340168 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.340228 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:40:22.341305 25077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:40:22.348548 25077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:40:22.384765 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.360408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:38445 (local address 127.0.0.1:42100)
0926 16:40:22.371867 (+ 11459us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.371886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:22.372002 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:22.374544 (+ 2542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:22.374552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:22.374620 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:22.375309 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.375326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.377575 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.377581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.380382 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.380400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.383666 (+ 3266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.383672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.383928 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.383981 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 16:40:22.384124 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 16:40:22.384345 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11306,"mutex_wait_us":1,"thread_start_us":10354,"threads_started":1}
I20260926 16:40:22.385316 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.361093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.10.110.2:38445)
0926 16:40:22.373822 (+ 12729us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.373832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.373870 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:40:22.373979 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.373988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.374293 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.374502 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:22.376274 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.377409 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.380584 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.383438 (+ 2854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.383561 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.383778 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.384331 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.384516 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12536,"thread_start_us":12403,"threads_started":1}
I20260926 16:40:22.388010 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.360071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58184)
0926 16:40:22.371319 (+ 11248us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.371389 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:22.371646 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:22.380708 (+ 9062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:22.380724 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:22.380764 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:22.381139 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.381153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.383304 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.383309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.385460 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.385480 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.387227 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.387234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.387410 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.387442 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:40:22.387537 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:40:22.387663 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10870,"thread_start_us":21211,"threads_started":1}
I20260926 16:40:22.388113 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.360185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.10.110.1:33019)
0926 16:40:22.379528 (+ 19343us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.379538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.379570 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:40:22.379677 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.379686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.380015 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.381346 (+ 1331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:22.381582 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.383151 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.385673 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.386995 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.387136 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.387321 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.387592 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.387756 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19155,"thread_start_us":19984,"threads_started":1}
I20260926 16:40:22.389299 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.361724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38651 (local address 127.0.0.1:33420)
0926 16:40:22.381034 (+ 19310us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.381055 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:22.381203 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:22.381898 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:22.381902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:22.381923 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:22.382221 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.382228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.384197 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.384202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.386571 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.386585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.388582 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.388587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.388703 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.388725 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:40:22.388827 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:40:22.388944 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19148,"thread_start_us":10124,"threads_started":1}
I20260926 16:40:22.389282 25330 tablet_service.cc:1467] Processing CreateTablet for tablet 7abbcf97094d43539a5f9490faebb26d (DEFAULT_TABLE table=test-table [id=078bbeaed73a42588bd1dd05f0f599d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:22.390015 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.360942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.10.110.3:38651)
0926 16:40:22.374017 (+ 13075us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.374024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.381184 (+ 7160us) server_negotiation.cc:407] Connection header received
0926 16:40:22.381408 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.381417 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.381703 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.381918 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:22.382403 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.383996 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.386743 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.388111 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.388234 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.388450 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.389562 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.389733 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12903,"thread_start_us":13073,"threads_started":1}
I20260926 16:40:22.390573 25330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abbcf97094d43539a5f9490faebb26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:22.392596 25198 tablet_service.cc:1467] Processing CreateTablet for tablet 7abbcf97094d43539a5f9490faebb26d (DEFAULT_TABLE table=test-table [id=078bbeaed73a42588bd1dd05f0f599d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:22.393767 25198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abbcf97094d43539a5f9490faebb26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:22.393633 25461 tablet_service.cc:1467] Processing CreateTablet for tablet 7abbcf97094d43539a5f9490faebb26d (DEFAULT_TABLE table=test-table [id=078bbeaed73a42588bd1dd05f0f599d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:22.394969 25461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7abbcf97094d43539a5f9490faebb26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:22.412132 25542 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Bootstrap starting.
I20260926 16:40:22.412873 25543 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Bootstrap starting.
I20260926 16:40:22.413843 25544 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Bootstrap starting.
I20260926 16:40:22.417478 25543 tablet_bootstrap.cc:654] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:22.417478 25542 tablet_bootstrap.cc:654] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:22.418289 25544 tablet_bootstrap.cc:654] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:22.418753 25543 log.cc:824] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:22.419556 25542 log.cc:824] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:22.419556 25544 log.cc:824] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:22.421768 25543 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: No bootstrap required, opened a new log
I20260926 16:40:22.422011 25543 ts_tablet_manager.cc:1397] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 16:40:22.422288 25544 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: No bootstrap required, opened a new log
I20260926 16:40:22.422546 25544 ts_tablet_manager.cc:1397] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 16:40:22.423211 25542 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: No bootstrap required, opened a new log
I20260926 16:40:22.423460 25542 ts_tablet_manager.cc:1397] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260926 16:40:22.429214 25543 raft_consensus.cc:348] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.429647 25543 raft_consensus.cc:374] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:22.429772 25543 raft_consensus.cc:729] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412834bd8a434356ba5bcb75d66c0826, State: Initialized, Role: FOLLOWER
I20260926 16:40:22.430347 25543 consensus_queue.cc:260] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.431335 25263 heartbeater.cc:503] Master 127.10.110.62:36663 was elected leader, sending a full tablet report...
I20260926 16:40:22.432021 25543 ts_tablet_manager.cc:1428] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
I20260926 16:40:22.432705 25544 raft_consensus.cc:348] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.433105 25544 raft_consensus.cc:374] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:22.433203 25544 raft_consensus.cc:729] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac387ee061246a396cf23f246551091, State: Initialized, Role: FOLLOWER
I20260926 16:40:22.433444 25542 raft_consensus.cc:348] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.433832 25542 raft_consensus.cc:374] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:22.433923 25542 raft_consensus.cc:729] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e48311d4b0d4d8793b09d5c020908ff, State: Initialized, Role: FOLLOWER
I20260926 16:40:22.433779 25544 consensus_queue.cc:260] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.434482 25542 consensus_queue.cc:260] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.440553 25526 heartbeater.cc:503] Master 127.10.110.62:36663 was elected leader, sending a full tablet report...
I20260926 16:40:22.440559 25395 heartbeater.cc:503] Master 127.10.110.62:36663 was elected leader, sending a full tablet report...
I20260926 16:40:22.441318 25544 ts_tablet_manager.cc:1428] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
I20260926 16:40:22.442857 25542 ts_tablet_manager.cc:1428] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Time spent starting tablet: real 0.019s user 0.013s sys 0.006s
I20260926 16:40:22.510929 25548 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:22.511220 25548 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.513010 25548 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
I20260926 16:40:22.519554 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.513473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48071 (local address 127.10.110.2:38445)
0926 16:40:22.513669 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.513675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.514347 (+ 672us) server_negotiation.cc:407] Connection header received
0926 16:40:22.514534 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.514539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.514598 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.514686 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:22.515301 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.516018 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.517872 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.518642 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.518783 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.518860 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.519170 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.519311 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:40:22.519564 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.513465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:53013 (local address 127.10.110.3:38651)
0926 16:40:22.513667 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.513673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.514422 (+ 749us) server_negotiation.cc:407] Connection header received
0926 16:40:22.514615 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.514619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.514677 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.514747 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:22.515348 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.516110 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.517901 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.518690 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.518834 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.518907 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.519209 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.519330 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:40:22.519706 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.513456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:38445 (local address 127.10.110.1:48071)
0926 16:40:22.514236 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.514252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:22.514384 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:22.514705 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:22.514709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:22.514748 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:22.515153 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.515166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.516159 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.516164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.517730 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.517743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.518861 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.518867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.519018 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.519043 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:40:22.519125 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 16:40:22.519266 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":36,"thread_start_us":413,"threads_started":1}
I20260926 16:40:22.519737 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.513515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38651 (local address 127.10.110.1:53013)
0926 16:40:22.514324 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.514341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:22.514471 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:22.514770 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:22.514774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:22.514795 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:22.515183 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.515191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.516249 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.516253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.517761 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:22.517773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.518901 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.518907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:22.519059 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.519080 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:40:22.519164 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:40:22.519292 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":22,"thread_start_us":442,"threads_started":1}
I20260926 16:40:22.520780 25350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abbcf97094d43539a5f9490faebb26d" candidate_uuid: "412834bd8a434356ba5bcb75d66c0826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e48311d4b0d4d8793b09d5c020908ff" is_pre_election: true
I20260926 16:40:22.520779 25481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abbcf97094d43539a5f9490faebb26d" candidate_uuid: "412834bd8a434356ba5bcb75d66c0826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac387ee061246a396cf23f246551091" is_pre_election: true
I20260926 16:40:22.521250 25481 raft_consensus.cc:2454] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412834bd8a434356ba5bcb75d66c0826 in term 0.
I20260926 16:40:22.521255 25350 raft_consensus.cc:2454] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 412834bd8a434356ba5bcb75d66c0826 in term 0.
I20260926 16:40:22.521860 25154 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826, 4e48311d4b0d4d8793b09d5c020908ff; no voters:
I20260926 16:40:22.522241 25548 raft_consensus.cc:2781] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:40:22.522360 25548 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:40:22.522455 25548 raft_consensus.cc:3037] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:22.524712 25548 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.525274 25548 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 1 election: Requested vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
I20260926 16:40:22.525846 25350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abbcf97094d43539a5f9490faebb26d" candidate_uuid: "412834bd8a434356ba5bcb75d66c0826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e48311d4b0d4d8793b09d5c020908ff"
I20260926 16:40:22.525846 25481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7abbcf97094d43539a5f9490faebb26d" candidate_uuid: "412834bd8a434356ba5bcb75d66c0826" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac387ee061246a396cf23f246551091"
I20260926 16:40:22.526067 25350 raft_consensus.cc:3037] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:22.526072 25481 raft_consensus.cc:3037] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:22.529250 25350 raft_consensus.cc:2454] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412834bd8a434356ba5bcb75d66c0826 in term 1.
I20260926 16:40:22.529250 25481 raft_consensus.cc:2454] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 412834bd8a434356ba5bcb75d66c0826 in term 1.
I20260926 16:40:22.529708 25152 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826, aac387ee061246a396cf23f246551091; no voters:
I20260926 16:40:22.530035 25548 raft_consensus.cc:2781] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:22.530480 25548 raft_consensus.cc:686] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 LEADER]: Becoming Leader. State: Replica: 412834bd8a434356ba5bcb75d66c0826, State: Running, Role: LEADER
I20260926 16:40:22.530987 25548 consensus_queue.cc:237] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:22.535264 25077 catalog_manager.cc:5495] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 reported cstate change: term changed from 0 to 1, leader changed from <none> to 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "412834bd8a434356ba5bcb75d66c0826" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } health_report { overall_health: UNKNOWN } } }
W20260926 16:40:22.558322 25527 tablet.cc:2369] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:22.581907 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.576859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.10.110.1:33019)
0926 16:40:22.577053 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.577057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.577077 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:40:22.577166 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.577170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.577254 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.577377 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:40:22.577698 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.578397 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.579822 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.580280 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.580376 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.580810 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:40:22.581328 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:40:22.581440 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.581585 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.581659 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:40:22.590021 25350 raft_consensus.cc:1261] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Refusing update from remote peer 412834bd8a434356ba5bcb75d66c0826: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:22.590021 25481 raft_consensus.cc:1261] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Refusing update from remote peer 412834bd8a434356ba5bcb75d66c0826: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:22.590973 25548 consensus_queue.cc:1035] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:22.591333 25553 consensus_queue.cc:1035] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:22.602521 25556 mvcc.cc:205] Tried to move back new op lower bound from 7333645609318793216 to 7333645609091002368. Current Snapshot: MvccSnapshot[applied={T|T < 7333645609318793216}]
I20260926 16:40:22.603124 25557 mvcc.cc:205] Tried to move back new op lower bound from 7333645609318793216 to 7333645609091002368. Current Snapshot: MvccSnapshot[applied={T|T < 7333645609318793216}]
I20260926 16:40:22.616317 25078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:22.616600 25078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:22.620363 25078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Sending DeleteTablet for 3 replicas of tablet 7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:22.621551 25461 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abbcf97094d43539a5f9490faebb26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:22 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:33420
I20260926 16:40:22.621559 25198 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abbcf97094d43539a5f9490faebb26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:22 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:58184
I20260926 16:40:22.621593 25330 tablet_service.cc:1514] Processing DeleteTablet for tablet 7abbcf97094d43539a5f9490faebb26d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:22 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:42100
I20260926 16:40:22.625085 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25115
I20260926 16:40:22.651387 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.626666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.10.110.1:33019)
0926 16:40:22.626872 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.626879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.626899 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:40:22.626964 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.626970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.627063 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.627160 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:22.629423 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.630795 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.633118 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.633860 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.633985 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.639764 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:22.639792 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:22.639882 (+ 90us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:22.639965 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:22.644532 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.647075 (+ 2543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.647087 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.647091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.647335 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.648322 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.648328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.648333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.648864 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:22.649259 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.650665 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.650934 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:40:22.653425 25258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:22.658679 25566 tablet_replica.cc:330] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: stopping tablet replica
I20260926 16:40:22.658914 25568 tablet_replica.cc:330] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: stopping tablet replica
I20260926 16:40:22.659431 25568 raft_consensus.cc:2229] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:22.660432 25568 raft_consensus.cc:2258] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:22.660596 25566 raft_consensus.cc:2229] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:22.661557 25566 raft_consensus.cc:2258] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:22.662554 25568 ts_tablet_manager.cc:1905] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:22.664608 25566 ts_tablet_manager.cc:1905] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:22.669354 25568 ts_tablet_manager.cc:1918] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:22.669561 25568 log.cc:1196] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:22.670105 25568 ts_tablet_manager.cc:1939] T 7abbcf97094d43539a5f9490faebb26d P 4e48311d4b0d4d8793b09d5c020908ff: Deleting consensus metadata
I20260926 16:40:22.671288 25566 ts_tablet_manager.cc:1918] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:22.671437 25566 log.cc:1196] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:22.671964 25566 ts_tablet_manager.cc:1939] T 7abbcf97094d43539a5f9490faebb26d P aac387ee061246a396cf23f246551091: Deleting consensus metadata
I20260926 16:40:22.672079 25065 catalog_manager.cc:4841] TS 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): tablet 7abbcf97094d43539a5f9490faebb26d (table test-table [id=078bbeaed73a42588bd1dd05f0f599d8]) successfully deleted
I20260926 16:40:22.673666 25063 catalog_manager.cc:4841] TS aac387ee061246a396cf23f246551091 (127.10.110.3:38651): tablet 7abbcf97094d43539a5f9490faebb26d (table test-table [id=078bbeaed73a42588bd1dd05f0f599d8]) successfully deleted
I20260926 16:40:22.769872 25567 tablet_replica.cc:330] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: stopping tablet replica
I20260926 16:40:22.770411 25567 raft_consensus.cc:2229] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:40:22.771188 25567 raft_consensus.cc:2258] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:22.771612 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25115
I20260926 16:40:22.787199 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25267
W20260926 16:40:22.787254 25062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.110.1:33019 (error 108)
I20260926 16:40:22.788960 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.788582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58204)
0926 16:40:22.788732 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.788825 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:40:22.789152 25062 catalog_manager.cc:4584] TS 412834bd8a434356ba5bcb75d66c0826 (127.10.110.1:33019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7abbcf97094d43539a5f9490faebb26d: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
I20260926 16:40:22.805310 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.787930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.10.110.2:38445)
0926 16:40:22.788148 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.788154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.788220 (+ 66us) server_negotiation.cc:407] Connection header received
0926 16:40:22.788419 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.788424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.788497 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.788579 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:22.789997 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.790776 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.792288 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.792727 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.792837 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.800142 (+ 7305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:22.800157 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:22.800174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:22.800217 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:22.802640 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.803278 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.803286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.803288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.803416 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.803792 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.803797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.803799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.804222 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:22.804483 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.804770 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.804970 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:40:22.805912 25390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:22.842733 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.842221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58206)
0926 16:40:22.842458 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.842560 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:40:22.843739 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.843395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58212)
0926 16:40:22.843538 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.843613 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 16:40:22.895193 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25267
I20260926 16:40:22.904127 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.903690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58222)
0926 16:40:22.903899 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.903983 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 16:40:22.905120 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.904816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58224)
0926 16:40:22.904937 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.905007 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 16:40:22.908948 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25398
I20260926 16:40:22.921980 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.909616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.10.110.3:38651)
0926 16:40:22.909791 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:22.909796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:22.909815 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:40:22.909957 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:22.909961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:22.910022 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:22.910126 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:22.911638 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.912387 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.913709 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:22.914109 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:22.914216 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:22.916769 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:22.916785 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:22.916802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:22.916838 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:22.919317 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.920105 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.920113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.920116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.920330 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:22.920685 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:22.920712 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:22.920715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:22.921064 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:22.921288 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:22.921476 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 16:40:22.921649 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:40:22.922571 25521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:22.985946 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.985364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58228)
0926 16:40:22.985617 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.985729 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 16:40:22.987143 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:22.986734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:33019 (local address 127.0.0.1:58242)
0926 16:40:22.986925 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:22.987011 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:40:23.020447 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25398
I20260926 16:40:23.034516 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25045
I20260926 16:40:23.035718 25108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:23.150379 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25045
I20260926 16:40:23.168371 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36663
--webserver_interface=127.10.110.62
--webserver_port=34485
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:23.491219 25574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:23.491513 25574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:23.491652 25574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:23.498972 25574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:23.499084 25574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:23.499156 25574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:23.499233 25574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:23.499286 25574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:23.509162 25574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=34485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:23.510504 25574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:23.511782 25574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:23.523046 25579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:23.523039 25580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:23.523074 25582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:23.523789 25574 server_base.cc:1034] running on GCE node
I20260926 16:40:23.524533 25574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:23.525856 25574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:23.527098 25574 hybrid_clock.cc:648] HybridClock initialized: now 1790440823527026 us; error 38 us; skew 500 ppm
I20260926 16:40:23.527448 25574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:23 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440823, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:23.530356 25574 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:23.531741 25574 webserver.cc:460] Webserver started at http://127.10.110.62:34485/ using document root <none> and password file <none>
I20260926 16:40:23.532419 25574 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:23.532532 25574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:23.537611 25574 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 16:40:23.540445 25589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:23.541792 25574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:23.541949 25574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "1731653a8faf450691ff92da350052d7"
format_stamp: "Formatted at 2026-09-26 16:40:17 on dist-test-slave-whjp"
server_key: "af32c31a5e10a1eff892944e0c403cd7"
server_key_iv: "252746bc97ee7c8e5d134d36b4fe509b"
server_key_version: "encryptionkey@0"
I20260926 16:40:23.542874 25574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:23.573704 25574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:23.577513 25574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:23.577770 25574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:23.595067 25574 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:36663
I20260926 16:40:23.595088 25640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:36663 every 8 connection(s)
I20260926 16:40:23.596621 25574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:23.599046 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25574
I20260926 16:40:23.600032 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:33019
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:36663
--webserver_port=45317
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:23.605429 25641 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:40:23.609573 25641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap starting.
I20260926 16:40:23.629796 25641 log.cc:824] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:23.645555 25641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:40:23.646158 25641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7: Bootstrap complete.
I20260926 16:40:23.655293 25641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:23.656301 25641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1731653a8faf450691ff92da350052d7, State: Initialized, Role: FOLLOWER
I20260926 16:40:23.656891 25641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:23.657141 25641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:23.657255 25641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:23.657415 25641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:40:23.661300 25641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:23.661794 25641 leader_election.cc:302] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1731653a8faf450691ff92da350052d7; no voters:
I20260926 16:40:23.662416 25641 leader_election.cc:288] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:40:23.662552 25645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:40:23.664332 25645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [term 3 LEADER]: Becoming Leader. State: Replica: 1731653a8faf450691ff92da350052d7, State: Running, Role: LEADER
I20260926 16:40:23.664922 25645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } }
I20260926 16:40:23.665490 25641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:23.667390 25646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1731653a8faf450691ff92da350052d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } } }
I20260926 16:40:23.667424 25647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1731653a8faf450691ff92da350052d7. Latest consensus state: current_term: 3 leader_uuid: "1731653a8faf450691ff92da350052d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1731653a8faf450691ff92da350052d7" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 36663 } } }
I20260926 16:40:23.667865 25647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:23.667865 25646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1731653a8faf450691ff92da350052d7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:23.668766 25650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:23.685598 25650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:23.687611 25650 catalog_manager.cc:1259] Loaded cluster ID: 7a0f832428924585b7ddbd1c02f19757
I20260926 16:40:23.687788 25650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 16:40:23.690121 25650 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa3ffa789ad google::LogMessage::Fail() at ??:0
@ 0x7fa3ffa7b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa3ffa784de google::LogMessage::Flush() at ??:0
@ 0x7fa3ffa79a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa4154ab512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa4154a8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa415530610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa415530439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa4026c4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa4026c791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa4026c7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa4026a2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa40597c6db start_thread at ??:0
@ 0x7fa3fe91261f clone at ??:0
W20260926 16:40:24.035075 25643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:24.035359 25643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:24.035468 25643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:24.042866 25643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:24.042980 25643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:24.043102 25643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:40:24.053313 25643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:33019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=45317
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:24.054602 25643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:24.055987 25643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:24.068399 25680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:24.068423 25683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:24.068451 25681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:24.069659 25643 server_base.cc:1034] running on GCE node
I20260926 16:40:24.070214 25643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:24.071106 25643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:24.072295 25643 hybrid_clock.cc:648] HybridClock initialized: now 1790440824072236 us; error 52 us; skew 500 ppm
I20260926 16:40:24.072582 25643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:24 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440824, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:24.075453 25643 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:40:24.076812 25643 webserver.cc:460] Webserver started at http://127.10.110.1:45317/ using document root <none> and password file <none>
I20260926 16:40:24.077447 25643 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:24.077553 25643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:24.082620 25643 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:40:24.085316 25690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:24.086580 25643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:40:24.086735 25643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "412834bd8a434356ba5bcb75d66c0826"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "e543a4b9cbf26c339dc71c301285ef31"
server_key_iv: "c5cf5e6156e0d1e36d556c8e6b20e9c1"
server_key_version: "encryptionkey@0"
I20260926 16:40:24.087666 25643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:24.113793 25643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:24.117697 25643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:24.117949 25643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:24.118968 25643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:24.120826 25697 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 16:40:24.126547 25643 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 16:40:24.126668 25643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 16:40:24.126796 25643 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 16:40:24.130043 25643 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 16:40:24.130125 25643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260926 16:40:24.130478 25697 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Bootstrap starting.
I20260926 16:40:24.156868 25697 log.cc:824] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:24.164860 25643 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:33019
I20260926 16:40:24.164880 25804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:33019 every 8 connection(s)
I20260926 16:40:24.166194 25643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:40:24.169551 25697 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:40:24.170176 25697 tablet_bootstrap.cc:492] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Bootstrap complete.
I20260926 16:40:24.171000 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.168959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:41855)
0926 16:40:24.169823 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.170080 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":379,"threads_started":1}
I20260926 16:40:24.171352 25697 ts_tablet_manager.cc:1397] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Time spent bootstrapping tablet: real 0.041s user 0.029s sys 0.008s
W20260926 16:40:24.171657 25805 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:36663 (0 consecutive failures): Network error: Failed to ping master at 127.10.110.62:36663: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
I20260926 16:40:24.172514 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.172178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:37989)
0926 16:40:24.172329 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.172402 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 16:40:24.173547 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.173075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:33135)
0926 16:40:24.173246 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.173350 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:40:24.173820 25805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:40:24.176143 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25643
I20260926 16:40:24.176735 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:38445
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:36663
--webserver_port=42573
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:24.178390 25697 raft_consensus.cc:348] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:24.179311 25697 raft_consensus.cc:729] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412834bd8a434356ba5bcb75d66c0826, State: Initialized, Role: FOLLOWER
I20260926 16:40:24.179952 25697 consensus_queue.cc:260] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:24.181643 25697 ts_tablet_manager.cc:1428] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826: Time spent starting tablet: real 0.010s user 0.009s sys 0.004s
I20260926 16:40:24.181833 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.181359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:43353)
0926 16:40:24.181535 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.181639 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 16:40:24.182832 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.182492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:59885)
0926 16:40:24.182643 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.182710 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:40:24.501994 25809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:24.502266 25809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:24.502389 25809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:24.510139 25809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:24.510296 25809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:24.510492 25809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:40:24.520702 25809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:38445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=42573
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:24.522028 25809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:24.523375 25809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:24.536624 25817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:24.536690 25814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:24.536656 25815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:24.537411 25809 server_base.cc:1034] running on GCE node
I20260926 16:40:24.537971 25809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:24.538954 25809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:24.540143 25809 hybrid_clock.cc:648] HybridClock initialized: now 1790440824540089 us; error 44 us; skew 500 ppm
I20260926 16:40:24.540447 25809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:24 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440824, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:24.543409 25809 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:40:24.544859 25809 webserver.cc:460] Webserver started at http://127.10.110.2:42573/ using document root <none> and password file <none>
I20260926 16:40:24.545538 25809 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:24.545656 25809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:24.550972 25809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 16:40:24.553754 25824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:24.555153 25809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:40:24.555305 25809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "4e48311d4b0d4d8793b09d5c020908ff"
format_stamp: "Formatted at 2026-09-26 16:40:18 on dist-test-slave-whjp"
server_key: "ceeb4e3bcb7e8dc3489a1d158ddec0b0"
server_key_iv: "4ed932c1f6d2e16f501aa7bb09b13f2f"
server_key_version: "encryptionkey@0"
I20260926 16:40:24.556371 25809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:24.577625 25809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:24.581691 25809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:24.581957 25809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:24.582913 25809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:24.584273 25809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:24.584367 25809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:24.584473 25809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:24.584548 25809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:24.614558 25809 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:38445
I20260926 16:40:24.614624 25936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:38445 every 8 connection(s)
I20260926 16:40:24.615986 25809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:40:24.618839 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25809
I20260926 16:40:24.619508 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:38651
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:36663
--webserver_port=37223
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:40:24.621320 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.618956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:40517)
0926 16:40:24.619808 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.620125 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":296,"threads_started":1}
W20260926 16:40:24.622071 25937 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:36663 (0 consecutive failures): Network error: Failed to ping master at 127.10.110.62:36663: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
I20260926 16:40:24.622972 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.622600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:44803)
0926 16:40:24.622758 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.622841 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 16:40:24.623884 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:24.623502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:45515)
0926 16:40:24.623638 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:24.623726 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 16:40:24.624166 25937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:40:24.953890 25940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:24.954167 25940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:24.954288 25940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:24.961642 25940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:24.961777 25940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:24.961923 25940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:40:24.971725 25940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:38651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=37223
--tserver_master_addrs=127.10.110.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:24.973068 25940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:24.974346 25940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:24.987035 25945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:24.986944 25946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:24.987296 25948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:24.987957 25940 server_base.cc:1034] running on GCE node
I20260926 16:40:24.988546 25940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:24.989487 25940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:24.990706 25940 hybrid_clock.cc:648] HybridClock initialized: now 1790440824990657 us; error 37 us; skew 500 ppm
I20260926 16:40:24.991048 25940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:24 dist-test-slave-whjp krb5kdc[24493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440824, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:24.994105 25940 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:40:24.995661 25940 webserver.cc:460] Webserver started at http://127.10.110.3:37223/ using document root <none> and password file <none>
I20260926 16:40:24.996407 25940 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:24.996515 25940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:25.002374 25940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:40:25.005277 25955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:25.006678 25940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:25.006840 25940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "aac387ee061246a396cf23f246551091"
format_stamp: "Formatted at 2026-09-26 16:40:19 on dist-test-slave-whjp"
server_key: "2c3f53284c0a686ea5cccc97cb5c76f5"
server_key_iv: "15da77f441b82cf113ee036fa3ee8096"
server_key_version: "encryptionkey@0"
I20260926 16:40:25.007890 25940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:25.034035 25940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:25.037995 25940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:25.038250 25940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:25.039196 25940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:25.040557 25940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:25.040654 25940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:25.040740 25940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:25.040812 25940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:25.070858 25940 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38651
I20260926 16:40:25.070924 26067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38651 every 8 connection(s)
I20260926 16:40:25.072341 25940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:40:25.073777 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 25940
I20260926 16:40:25.076996 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.075226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:36387)
0926 16:40:25.075941 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.076148 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":259,"threads_started":1}
W20260926 16:40:25.077625 26068 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:36663 (0 consecutive failures): Network error: Failed to ping master at 127.10.110.62:36663: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
I20260926 16:40:25.078384 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.078086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:58053)
0926 16:40:25.078213 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.078281 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 16:40:25.079159 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.078852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:50283)
0926 16:40:25.078976 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.079057 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 16:40:25.079372 26068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:40:25.184525 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.183666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:47287)
0926 16:40:25.184202 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.184327 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":271,"threads_started":1}
I20260926 16:40:25.577653 26073 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:25.577958 26073 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:25.579710 26073 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
Sep 26 16:40:25 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440824, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
Sep 26 16:40:25 dist-test-slave-whjp krb5kdc[24493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440824, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:40:25.602587 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.580042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:38445 (local address 127.10.110.1:41023)
0926 16:40:25.580300 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.580346 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:25.580978 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:25.584354 (+ 3376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:25.584366 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:25.585095 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:25.585807 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:25.585827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.587320 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.587324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:25.588966 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:25.588980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.589960 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.589970 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:25.590231 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:25.591081 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:25.591098 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:25.593508 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:25.597904 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:25.597911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:25.597924 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:25.598742 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:25.599469 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:25.599474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:25.599477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:25.599763 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:25.600821 (+ 1058us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:25.600827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:25.601195 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 16:40:25.601524 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 16:40:25.602244 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":391,"spinlock_wait_cycles":35968}
I20260926 16:40:25.602620 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.580640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:36321 (local address 127.10.110.3:38651)
0926 16:40:25.582010 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:25.582024 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:25.582503 (+ 479us) server_negotiation.cc:407] Connection header received
0926 16:40:25.583987 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:25.583995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:25.584325 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:25.584585 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:25.585994 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.587146 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.589112 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.589757 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.589940 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:25.594018 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:25.594028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:25.594044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:25.594097 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:25.597731 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:25.599456 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:25.599464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:25.599472 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:25.599701 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:25.600245 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:25.600249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:25.600252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:25.600814 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:25.600976 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:25.601762 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 16:40:25.602196 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":1103,"threads_started":1}
I20260926 16:40:25.602620 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.580454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:41023 (local address 127.10.110.2:38445)
0926 16:40:25.581744 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:25.581761 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:25.581804 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:40:25.583416 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:25.583426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:25.583875 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:25.584197 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:25.586003 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.587142 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.589105 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.589756 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.589937 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:25.593805 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:25.593812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:25.593830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:25.593884 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:25.597738 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:25.598931 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:25.598940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:25.598948 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:25.599316 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:25.599930 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:25.599936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:25.599938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:25.600680 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:25.600894 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:25.601770 (+ 876us) server_negotiation.cc:299] Negotiation successful
0926 16:40:25.602196 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":950,"threads_started":1}
I20260926 16:40:25.602569 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.580518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38651 (local address 127.10.110.1:36321)
0926 16:40:25.582381 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.582402 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:25.582524 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:25.584591 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:25.584596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:25.585199 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:25.585807 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:25.585828 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.587327 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.587331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:25.588972 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:25.588984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:25.589998 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:25.590006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:25.590225 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:25.591047 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:25.591074 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:25.593768 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:25.598867 (+ 5099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:25.598870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:25.598873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:25.599305 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:25.599973 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:25.599977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:25.599980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:25.600128 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:25.600929 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:25.600933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:25.601195 (+ 262us) client_negotiation.cc:769] Sending connection context
0926 16:40:25.601525 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 16:40:25.602105 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"mutex_wait_us":177,"spinlock_wait_cycles":768,"thread_start_us":1641,"threads_started":1}
W20260926 16:40:25.606380 25692 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aac387ee061246a396cf23f246551091 (127.10.110.3:38651): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
W20260926 16:40:25.606654 25694 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:25.606808 25694 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826; no voters: 4e48311d4b0d4d8793b09d5c020908ff, aac387ee061246a396cf23f246551091
I20260926 16:40:25.607200 26073 raft_consensus.cc:2726] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:40:25.625572 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:25.624703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:47529)
0926 16:40:25.625182 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:25.625332 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260926 16:40:26.080730 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:26.079891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:55205)
0926 16:40:26.080319 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:26.080469 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":200,"threads_started":1}
I20260926 16:40:26.186076 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:26.185397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:54999)
0926 16:40:26.185821 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:26.185904 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":206,"threads_started":1}
I20260926 16:40:26.627148 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:26.626449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:35811)
0926 16:40:26.626907 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:26.626989 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260926 16:40:27.082360 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:27.081672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:57163)
0926 16:40:27.082115 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:27.082206 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260926 16:40:27.187739 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:27.186929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:54817)
0926 16:40:27.187432 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:27.187553 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":237,"threads_started":1}
I20260926 16:40:27.535930 26083 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:27.536129 26083 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:27.536757 26083 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
W20260926 16:40:27.537717 25692 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aac387ee061246a396cf23f246551091 (127.10.110.3:38651): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
W20260926 16:40:27.537884 25694 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:27.538008 25694 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826; no voters: 4e48311d4b0d4d8793b09d5c020908ff, aac387ee061246a396cf23f246551091
I20260926 16:40:27.538236 26083 raft_consensus.cc:2726] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:40:27.628814 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:27.628097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:58011)
0926 16:40:27.628556 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:27.628651 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260926 16:40:28.083961 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:28.083233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:36947)
0926 16:40:28.083700 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:28.083784 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":220,"threads_started":1}
I20260926 16:40:28.189328 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:28.188591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:56819)
0926 16:40:28.189067 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:28.189155 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260926 16:40:28.630507 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:28.629727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:52093)
0926 16:40:28.630223 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:28.630323 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":236,"threads_started":1}
I20260926 16:40:29.085659 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:29.084865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:40759)
0926 16:40:29.085328 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:29.085418 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":209,"threads_started":1}
I20260926 16:40:29.191241 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:29.190395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:50783)
0926 16:40:29.190905 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:29.191043 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260926 16:40:29.632539 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:29.631537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:34363)
0926 16:40:29.632169 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:29.632344 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":358,"threads_started":1}
I20260926 16:40:30.087442 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:30.086683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:44135)
0926 16:40:30.087193 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:30.087280 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":289,"threads_started":1}
I20260926 16:40:30.193207 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:30.192352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:47107)
0926 16:40:30.192916 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:30.193014 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":332,"threads_started":1}
I20260926 16:40:30.264088 26093 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:30.264302 26093 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:30.265061 26093 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
W20260926 16:40:30.266033 25694 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
W20260926 16:40:30.266240 25692 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aac387ee061246a396cf23f246551091 (127.10.110.3:38651): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:30.266384 25692 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826; no voters: 4e48311d4b0d4d8793b09d5c020908ff, aac387ee061246a396cf23f246551091
I20260926 16:40:30.266645 26093 raft_consensus.cc:2726] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:40:30.634526 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:30.633627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:35507)
0926 16:40:30.634235 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:30.634336 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":307,"threads_started":1}
I20260926 16:40:31.089139 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:31.088338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:55111)
0926 16:40:31.088847 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:31.088930 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":268,"threads_started":1}
I20260926 16:40:31.194985 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:31.194177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:39411)
0926 16:40:31.194678 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:31.194772 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":224,"threads_started":1}
I20260926 16:40:31.636210 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:31.635406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:34841)
0926 16:40:31.635934 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:31.636019 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":216,"threads_started":1}
I20260926 16:40:32.090864 26098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:32.090085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:52683)
0926 16:40:32.090573 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:32.090687 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":212,"threads_started":1}
I20260926 16:40:32.196655 26099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:32.195973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:39093)
0926 16:40:32.196418 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:32.196500 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":227,"threads_started":1}
I20260926 16:40:32.227700 26100 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:32.228048 26100 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:32.228734 26100 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
W20260926 16:40:32.229637 25694 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
W20260926 16:40:32.229806 25692 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aac387ee061246a396cf23f246551091 (127.10.110.3:38651): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:32.229928 25692 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826; no voters: 4e48311d4b0d4d8793b09d5c020908ff, aac387ee061246a396cf23f246551091
I20260926 16:40:32.230170 26100 raft_consensus.cc:2726] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:40:32.638126 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:32.637353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:38865)
0926 16:40:32.637836 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:32.637924 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260926 16:40:33.092655 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:33.091929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:59749)
0926 16:40:33.092381 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:33.092465 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260926 16:40:33.198262 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:33.197565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:60569)
0926 16:40:33.198010 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:33.198085 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":224,"threads_started":1}
I20260926 16:40:33.639919 26104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:33.639149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:57267)
0926 16:40:33.639639 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:33.639740 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":226,"threads_started":1}
I20260926 16:40:34.094136 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:34.093466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:53117)
0926 16:40:34.093898 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:34.093980 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":206,"threads_started":1}
I20260926 16:40:34.200004 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:34.199242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:43391)
0926 16:40:34.199701 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:34.199807 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":245,"threads_started":1}
I20260926 16:40:34.641682 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:34.640955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:46341)
0926 16:40:34.641412 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:34.641505 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260926 16:40:35.095752 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:35.095033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:48461)
0926 16:40:35.095480 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:35.095577 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":210,"threads_started":1}
I20260926 16:40:35.201844 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:35.200941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:38863)
0926 16:40:35.201446 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:35.201626 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260926 16:40:35.643406 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:35.642592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:48887)
0926 16:40:35.643138 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:35.643232 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":286,"threads_started":1}
I20260926 16:40:35.915874 26111 raft_consensus.cc:482] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:35.916078 26111 raft_consensus.cc:504] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "412834bd8a434356ba5bcb75d66c0826" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 33019 } } peers { permanent_uuid: "4e48311d4b0d4d8793b09d5c020908ff" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 38445 } } peers { permanent_uuid: "aac387ee061246a396cf23f246551091" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38651 } }
I20260926 16:40:35.916740 26111 leader_election.cc:288] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445), aac387ee061246a396cf23f246551091 (127.10.110.3:38651)
W20260926 16:40:35.917690 25694 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4e48311d4b0d4d8793b09d5c020908ff (127.10.110.2:38445): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
W20260926 16:40:35.917855 25692 leader_election.cc:341] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aac387ee061246a396cf23f246551091 (127.10.110.3:38651): Not found: Tablet not found: 7abbcf97094d43539a5f9490faebb26d
I20260926 16:40:35.917968 25692 leader_election.cc:302] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 412834bd8a434356ba5bcb75d66c0826; no voters: 4e48311d4b0d4d8793b09d5c020908ff, aac387ee061246a396cf23f246551091
I20260926 16:40:35.918195 26111 raft_consensus.cc:2726] T 7abbcf97094d43539a5f9490faebb26d P 412834bd8a434356ba5bcb75d66c0826 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:40:36.097394 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:36.096710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:50569)
0926 16:40:36.097150 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:36.097239 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":212,"threads_started":1}
I20260926 16:40:36.203612 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:36.202756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:45823)
0926 16:40:36.203251 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:36.203351 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
I20260926 16:40:36.645154 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:36.644350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:43699)
0926 16:40:36.644886 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:36.644976 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
I20260926 16:40:37.098994 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:37.098291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:36083)
0926 16:40:37.098744 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:37.098821 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":243,"threads_started":1}
I20260926 16:40:37.205303 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:37.204562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:34305)
0926 16:40:37.205014 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:37.205113 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20260926 16:40:37.646749 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:37.646032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:34761)
0926 16:40:37.646496 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:37.646588 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260926 16:40:38.100739 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:38.100015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:39519)
0926 16:40:38.100475 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:38.100569 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":224,"threads_started":1}
I20260926 16:40:38.206933 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:38.206133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:35675)
0926 16:40:38.206616 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:38.206712 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260926 16:40:38.648815 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:38.647891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:47515)
0926 16:40:38.648430 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:38.648539 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
I20260926 16:40:39.102351 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:39.101621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:34233)
0926 16:40:39.102071 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:39.102152 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":228,"threads_started":1}
I20260926 16:40:39.208809 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:39.208096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.1:51983)
0926 16:40:39.208523 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:39.208619 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":202,"threads_started":1}
I20260926 16:40:39.651341 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:39.650110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.2:45747)
0926 16:40:39.651032 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:39.651151 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":681,"threads_started":1}
I20260926 16:40:40.085269 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25643
I20260926 16:40:40.104172 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:40.103426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:36663 (local address 127.10.110.3:36955)
0926 16:40:40.103879 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:40.103994 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":195,"threads_started":1}
I20260926 16:40:40.112933 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:40.086455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.10.110.1:33019)
0926 16:40:40.087025 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:40.087036 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:40.087074 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:40:40.087603 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:40.087616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:40.087911 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:40.088091 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:40.089840 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:40.090895 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:40.092323 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:40.092913 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:40.093055 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:40.107336 (+ 14281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:40.107353 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:40.107369 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:40.107415 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:40.110049 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:40.110807 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:40.110815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:40.110817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:40.110974 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:40.111332 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:40.111336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:40.111338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:40.111706 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:40.112012 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:40.112371 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 16:40:40.112641 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":366,"threads_started":1}
I20260926 16:40:40.114198 25800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:40.201632 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25643
I20260926 16:40:40.216010 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25809
I20260926 16:40:40.228713 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:40.216710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.10.110.2:38445)
0926 16:40:40.217115 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:40.217120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:40.217136 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:40:40.217191 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:40.217195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:40.217257 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:40.217344 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:40.218638 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:40.219375 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:40.220658 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:40.221074 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:40.221163 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:40.223759 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:40.223764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:40.223767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:40.223801 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:40.226516 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:40.227104 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:40.227108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:40.227110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:40.227227 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:40.227494 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:40.227496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:40.227498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:40.227772 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:40.228033 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:40.228197 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 16:40:40.228368 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":201,"threads_started":1}
I20260926 16:40:40.229405 25932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:40.312070 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25809
I20260926 16:40:40.327194 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 25940
I20260926 16:40:40.340482 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:40.328239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.10.110.3:38651)
0926 16:40:40.328850 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:40.328855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:40.328872 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:40:40.328927 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:40.328930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:40.328988 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:40.329079 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:40.330399 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:40.331153 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:40.332371 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:40.332806 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:40.332900 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:40.335361 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:40.335366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:40.335370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:40.335400 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:40.337972 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:40.338638 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:40.338642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:40.338645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:40.338811 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:40.339164 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:40.339167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:40.339169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:40.339447 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:40.339654 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:40.339914 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 16:40:40.340126 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":356,"threads_started":1}
I20260926 16:40:40.341171 26063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:40.428020 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 25940
2026-09-26T16:40:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24158 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:40:40 dist-test-slave-whjp krb5kdc[26133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:40:40 dist-test-slave-whjp krb5kdc[26133](info): set up 2 sockets
Sep 26 16:40:40 dist-test-slave-whjp krb5kdc[26133](info): commencing operation
krb5kdc: starting...
W20260926 16:40:41.444989 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:40:41 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:40:41.670964 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34537
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:42.006587 26146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:42.006896 26146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:42.007014 26146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:42.014321 26146 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:40:42.014413 26146 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:40:42.014451 26146 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:40:42.014528 26146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:40:42.014598 26146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:42.014681 26146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:42.014731 26146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:40:42.014798 26146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:40:42.024760 26146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:42.026057 26146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:42.027421 26146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:42.038758 26151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:42.038654 26152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:42.038689 26154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:42.039850 26146 server_base.cc:1034] running on GCE node
I20260926 16:40:42.040627 26146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:42.042078 26146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:42.043319 26146 hybrid_clock.cc:648] HybridClock initialized: now 1790440842043253 us; error 32 us; skew 500 ppm
I20260926 16:40:42.043670 26146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:42 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440842, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:42.046805 26146 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:40:42.048543 26146 webserver.cc:460] Webserver started at http://127.10.110.62:40653/ using document root <none> and password file <none>
I20260926 16:40:42.049248 26146 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:42.049336 26146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:42.049636 26146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:42.052646 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "b054271b34104531b2d1bf3f4cb75e73"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "cb6451e9d9baf62d146bfd1bdea4b250"
server_key_iv: "2372400fdd724ed1a92b552278a76ae1"
server_key_version: "encryptionkey@0"
I20260926 16:40:42.053401 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "b054271b34104531b2d1bf3f4cb75e73"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "cb6451e9d9baf62d146bfd1bdea4b250"
server_key_iv: "2372400fdd724ed1a92b552278a76ae1"
server_key_version: "encryptionkey@0"
I20260926 16:40:42.058459 26146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:40:42.061939 26161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:42.063339 26146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:40:42.063517 26146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "b054271b34104531b2d1bf3f4cb75e73"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "cb6451e9d9baf62d146bfd1bdea4b250"
server_key_iv: "2372400fdd724ed1a92b552278a76ae1"
server_key_version: "encryptionkey@0"
I20260926 16:40:42.063733 26146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:42.078552 26146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:42.082641 26146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:42.082891 26146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:42.100392 26146 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:34537
I20260926 16:40:42.100411 26212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:34537 every 8 connection(s)
I20260926 16:40:42.101691 26146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:40:42.105325 26213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:42.111923 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26146
I20260926 16:40:42.112113 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:40:42.112756 10680 external_mini_cluster.cc:1428] Setting key e14e7bc3f390dc073e41d731f48e987a
I20260926 16:40:42.115926 26213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap starting.
I20260926 16:40:42.119333 26213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:42.120517 26213 log.cc:824] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:42.123690 26213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: No bootstrap required, opened a new log
Sep 26 16:40:42 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:42.132496 26213 raft_consensus.cc:348] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:40:42.132905 26213 raft_consensus.cc:374] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:42.133013 26213 raft_consensus.cc:729] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b054271b34104531b2d1bf3f4cb75e73, State: Initialized, Role: FOLLOWER
I20260926 16:40:42.133628 26213 consensus_queue.cc:260] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:40:42.133925 26213 raft_consensus.cc:388] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:40:42.134068 26213 raft_consensus.cc:482] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:40:42.134258 26213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:42.136679 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:42.114782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.10.110.62:34537)
0926 16:40:42.115538 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:42.115549 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:42.115589 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:40:42.117363 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:42.117389 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:42.117802 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:42.118116 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:42.119906 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:42.121437 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:42.123266 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:42.124015 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:42.124153 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:42.128777 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:42.128786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:42.128806 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:42.128852 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:42.131988 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:42.132709 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:42.132723 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:42.132730 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:42.132926 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:42.133381 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:42.133386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:42.133389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:42.134120 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:42.134429 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:42.134805 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 16:40:42.135289 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":349,"threads_started":1}
I20260926 16:40:42.137405 26213 raft_consensus.cc:504] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:40:42.137909 26213 leader_election.cc:302] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b054271b34104531b2d1bf3f4cb75e73; no voters:
I20260926 16:40:42.138511 26213 leader_election.cc:288] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:40:42.138635 26218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:42.139689 26218 raft_consensus.cc:686] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 LEADER]: Becoming Leader. State: Replica: b054271b34104531b2d1bf3f4cb75e73, State: Running, Role: LEADER
I20260926 16:40:42.140332 26218 consensus_queue.cc:237] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:40:42.140964 26213 sys_catalog.cc:564] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:40:42.143414 26220 sys_catalog.cc:455] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b054271b34104531b2d1bf3f4cb75e73. Latest consensus state: current_term: 1 leader_uuid: "b054271b34104531b2d1bf3f4cb75e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } } }
I20260926 16:40:42.143200 26219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b054271b34104531b2d1bf3f4cb75e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } } }
I20260926 16:40:42.143850 26219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:42.143977 26220 sys_catalog.cc:458] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: This master's current role is: LEADER
I20260926 16:40:42.148473 26226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:40:42.153173 26226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:40:42.161940 26226 catalog_manager.cc:1347] Generated new cluster ID: f03eda8b4da048fb9a0ce1006cfcbd51
I20260926 16:40:42.162056 26226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:40:42.189495 26226 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:40:42.190891 26226 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:40:42.198305 26226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Generated new TSK 0
I20260926 16:40:42.199153 26226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:40:42.263058 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34537
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:42.610599 26239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:42.610868 26239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:42.610958 26239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:42.618002 26239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:42.618127 26239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:42.618305 26239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:40:42.628126 26239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:42.629684 26239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:42.631110 26239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:42.644538 26244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:42.644501 26245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:42.644565 26247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:42.645378 26239 server_base.cc:1034] running on GCE node
I20260926 16:40:42.646015 26239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:42.646993 26239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:42.648201 26239 hybrid_clock.cc:648] HybridClock initialized: now 1790440842648132 us; error 54 us; skew 500 ppm
I20260926 16:40:42.648525 26239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:42 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440842, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:42.652010 26239 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:40:42.653750 26239 webserver.cc:460] Webserver started at http://127.10.110.1:34867/ using document root <none> and password file <none>
I20260926 16:40:42.654584 26239 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:42.654698 26239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:42.655011 26239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:42.658026 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "b3cf59495051424a8db92217b2ace2fa"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "674c6f5cb1e7ed0e1e278a8a4ce892a4"
server_key_iv: "3ab3491d5100aec63fd3594be55b2425"
server_key_version: "encryptionkey@0"
I20260926 16:40:42.658819 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "b3cf59495051424a8db92217b2ace2fa"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "674c6f5cb1e7ed0e1e278a8a4ce892a4"
server_key_iv: "3ab3491d5100aec63fd3594be55b2425"
server_key_version: "encryptionkey@0"
I20260926 16:40:42.664295 26239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 16:40:42.668197 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:42.669739 26239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:40:42.669891 26239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "b3cf59495051424a8db92217b2ace2fa"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "674c6f5cb1e7ed0e1e278a8a4ce892a4"
server_key_iv: "3ab3491d5100aec63fd3594be55b2425"
server_key_version: "encryptionkey@0"
I20260926 16:40:42.670085 26239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:42.707523 26239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:42.711927 26239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:42.712199 26239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:42.713150 26239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:42.714505 26239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:42.714581 26239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:42.714687 26239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:42.714730 26239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:42.746184 26239 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:45267
I20260926 16:40:42.746237 26366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:45267 every 8 connection(s)
I20260926 16:40:42.747442 26239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:40:42.750876 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26239
I20260926 16:40:42.751104 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:40:42.751940 10680 external_mini_cluster.cc:1428] Setting key 4d6645769bcdc724340da0a066c2b88e
Sep 26 16:40:42 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440842, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:42.767139 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:42.750064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:51949 (local address 127.10.110.62:34537)
0926 16:40:42.750546 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:42.750551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:42.751666 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 16:40:42.753558 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:42.753564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:42.753639 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:42.753739 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:42.755287 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:42.756078 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:42.757653 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:42.758057 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:42.758172 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:42.761702 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:42.761714 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:42.761718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:42.761752 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:42.764157 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:42.764870 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:42.764875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:42.764877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:42.764996 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:42.765414 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:42.765419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:42.765421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:42.765684 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:42.765824 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:42.766548 (+ 724us) server_negotiation.cc:299] Negotiation successful
0926 16:40:42.766788 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":277,"threads_started":1}
I20260926 16:40:42.768184 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:42.750633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:51949)
0926 16:40:42.751499 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:42.751538 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:42.753292 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:42.753988 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:42.753996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:42.754555 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:42.755109 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:42.755128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:42.756245 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:42.756250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:42.757506 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:42.757519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:42.758294 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:42.758301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:42.758534 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:42.759157 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:42.759187 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:42.761479 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:42.764298 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:42.764307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:42.764327 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:42.764749 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:42.765101 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:42.765107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:42.765110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:42.765308 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:42.765814 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:42.765822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:42.766166 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 16:40:42.766418 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 16:40:42.766956 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":370,"threads_started":1}
I20260926 16:40:42.769819 26367 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34537
I20260926 16:40:42.770115 26367 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:42.770855 26367 heartbeater.cc:511] Master 127.10.110.62:34537 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:40:42.773116 26178 ts_manager.cc:194] Registered new tserver with Master: b3cf59495051424a8db92217b2ace2fa (127.10.110.1:45267)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:40:42.774966 26178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:51949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:40:42.801434 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34537
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:43.122874 26373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:43.123164 26373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:43.123258 26373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:43.130646 26373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:43.130779 26373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:43.130931 26373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:40:43.140951 26373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:43.142302 26373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:43.143618 26373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:43.156800 26378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:43.156911 26381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:43.156862 26379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:43.157785 26373 server_base.cc:1034] running on GCE node
I20260926 16:40:43.158412 26373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:43.159372 26373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:43.160570 26373 hybrid_clock.cc:648] HybridClock initialized: now 1790440843160512 us; error 49 us; skew 500 ppm
I20260926 16:40:43.160878 26373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:43 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440843, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:43.163986 26373 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:40:43.165421 26373 webserver.cc:460] Webserver started at http://127.10.110.2:35857/ using document root <none> and password file <none>
I20260926 16:40:43.166100 26373 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:43.166215 26373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:43.166491 26373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:43.169538 26373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "363ce0165ab346b68cb4bb49e48bf529"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "1a85bcc21ee26664e9bdadcaa7659653"
server_key_iv: "30304473f96b3da3fade3ee0c690bfc1"
server_key_version: "encryptionkey@0"
I20260926 16:40:43.170310 26373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "363ce0165ab346b68cb4bb49e48bf529"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "1a85bcc21ee26664e9bdadcaa7659653"
server_key_iv: "30304473f96b3da3fade3ee0c690bfc1"
server_key_version: "encryptionkey@0"
I20260926 16:40:43.175625 26373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:40:43.179136 26388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:43.180688 26373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:43.180869 26373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "363ce0165ab346b68cb4bb49e48bf529"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "1a85bcc21ee26664e9bdadcaa7659653"
server_key_iv: "30304473f96b3da3fade3ee0c690bfc1"
server_key_version: "encryptionkey@0"
I20260926 16:40:43.181084 26373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:43.201819 26373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:43.206218 26373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:43.206595 26373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:43.207620 26373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:43.209048 26373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:43.209131 26373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:43.209259 26373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:43.209312 26373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:43.240039 26373 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:46421
I20260926 16:40:43.240082 26500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:46421 every 8 connection(s)
I20260926 16:40:43.241343 26373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:40:43.243090 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26373
I20260926 16:40:43.243268 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:40:43.243868 10680 external_mini_cluster.cc:1428] Setting key 30af96e834c84c4ec39787e08d4fbc79
Sep 26 16:40:43 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440843, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:43.263583 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.244099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:38881 (local address 127.10.110.62:34537)
0926 16:40:43.244278 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:43.244282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:43.245841 (+ 1559us) server_negotiation.cc:407] Connection header received
0926 16:40:43.247697 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:43.247704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:43.247795 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:43.247913 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:43.249558 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.250594 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.252305 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.252924 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.253027 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.257245 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:43.257253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:43.257258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:43.257302 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:43.260370 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:43.261166 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:43.261172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:43.261175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:43.261341 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:43.261769 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:43.261773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:43.261776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:43.262116 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:43.262258 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:43.262993 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 16:40:43.263249 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:40:43.264528 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.244760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:38881)
0926 16:40:43.245665 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:43.245711 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:43.247406 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:43.248111 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:43.248119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:43.248718 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:43.249347 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.249366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.250757 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.250761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.252161 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.252174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.253191 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.253199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.253512 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.254264 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:43.254297 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:43.256954 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:43.260540 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:43.260549 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:43.260565 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:43.261035 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:43.261456 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:43.261459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:43.261462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:43.261656 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:43.262222 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:43.262228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:43.262581 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 16:40:43.262847 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 16:40:43.263323 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":352,"threads_started":1}
I20260926 16:40:43.266106 26501 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34537
I20260926 16:40:43.266419 26501 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:43.267140 26501 heartbeater.cc:511] Master 127.10.110.62:34537 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:40:43.268770 26178 ts_manager.cc:194] Registered new tserver with Master: 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421)
I20260926 16:40:43.269642 26178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:38881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:40:43.299944 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34537
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:40:43.648344 26506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:40:43.648691 26506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:40:43.648828 26506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:40:43.656361 26506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:40:43.656482 26506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:40:43.656633 26506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:40:43.666620 26506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:40:43.667938 26506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:40:43.669261 26506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:40:43.681747 26512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:43.681767 26514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:40:43.681777 26511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:40:43.685259 26506 server_base.cc:1034] running on GCE node
I20260926 16:40:43.685907 26506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:40:43.686806 26506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:40:43.688010 26506 hybrid_clock.cc:648] HybridClock initialized: now 1790440843687943 us; error 52 us; skew 500 ppm
I20260926 16:40:43.688344 26506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:40:43 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440843, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:40:43.691390 26506 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:40:43.692813 26506 webserver.cc:460] Webserver started at http://127.10.110.3:40925/ using document root <none> and password file <none>
I20260926 16:40:43.693526 26506 fs_manager.cc:359] Metadata directory not provided
I20260926 16:40:43.693624 26506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:40:43.693871 26506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:40:43.696784 26506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "0d56d0f8839b4147a39b4293c8480238"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "5a38c07d7d37717e4d19f32f2fc4dd4c"
server_key_iv: "d6140bed81b4c3d60c00c86044717ed4"
server_key_version: "encryptionkey@0"
I20260926 16:40:43.697477 26506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "0d56d0f8839b4147a39b4293c8480238"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "5a38c07d7d37717e4d19f32f2fc4dd4c"
server_key_iv: "d6140bed81b4c3d60c00c86044717ed4"
server_key_version: "encryptionkey@0"
I20260926 16:40:43.705540 26506 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260926 16:40:43.710984 26521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:43.712533 26506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:40:43.712682 26506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "0d56d0f8839b4147a39b4293c8480238"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "5a38c07d7d37717e4d19f32f2fc4dd4c"
server_key_iv: "d6140bed81b4c3d60c00c86044717ed4"
server_key_version: "encryptionkey@0"
I20260926 16:40:43.712872 26506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:40:43.738618 26506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:40:43.743243 26506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:40:43.743511 26506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:40:43.744489 26506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:40:43.745878 26506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:40:43.745970 26506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:43.746085 26506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:40:43.746138 26506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:40:43.775552 26506 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38107
I20260926 16:40:43.775600 26633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38107 every 8 connection(s)
I20260926 16:40:43.776829 26506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:40:43.778456 26367 heartbeater.cc:503] Master 127.10.110.62:34537 was elected leader, sending a full tablet report...
I20260926 16:40:43.783146 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26506
I20260926 16:40:43.783347 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:40:43.783922 10680 external_mini_cluster.cc:1428] Setting key 7012ea57571d5b546733d90505eef766
Sep 26 16:40:43 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440843, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:40:43.797812 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.779425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:53607 (local address 127.10.110.62:34537)
0926 16:40:43.779948 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:43.779954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:43.780986 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 16:40:43.782751 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:43.782756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:43.782823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:43.782914 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:43.784875 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.785778 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.788175 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.788605 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.788713 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.792028 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:43.792035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:43.792040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:43.792077 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:43.794496 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:43.795293 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:43.795298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:43.795301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:43.795468 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:43.795990 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:43.795994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:43.795996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:43.796315 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:43.796436 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:43.797254 (+ 818us) server_negotiation.cc:299] Negotiation successful
0926 16:40:43.797463 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
I20260926 16:40:43.798655 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.780099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:53607)
0926 16:40:43.780841 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:43.780884 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:43.782509 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:43.783180 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:43.783192 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:43.783944 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:40:43.784656 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.784674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.786586 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.786591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.787968 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.787986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.788798 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.788803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.789005 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.789632 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:40:43.789655 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:40:43.791748 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:40:43.794642 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:43.794650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:43.794666 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:43.795113 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:43.795605 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:40:43.795609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:40:43.795611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:40:43.795863 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:40:43.796427 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:40:43.796434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:40:43.796800 (+ 366us) client_negotiation.cc:769] Sending connection context
0926 16:40:43.797032 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 16:40:43.797459 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":293,"threads_started":1}
I20260926 16:40:43.800006 26634 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34537
I20260926 16:40:43.800287 26634 heartbeater.cc:464] Registering TS with master...
I20260926 16:40:43.800949 26634 heartbeater.cc:511] Master 127.10.110.62:34537 requested a full tablet report, sending...
I20260926 16:40:43.802304 26178 ts_manager.cc:194] Registered new tserver with Master: 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107)
I20260926 16:40:43.803107 26178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:53607
I20260926 16:40:43.811995 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:40:43.829036 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.816558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.10.110.62:34537)
0926 16:40:43.816734 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:43.816739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:43.817145 (+ 406us) server_negotiation.cc:407] Connection header received
0926 16:40:43.817302 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:43.817306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:43.817385 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:43.817473 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:43.818800 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.819515 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.820860 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.821267 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.821366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.823584 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:43.823589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:43.823593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:43.823627 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:43.826272 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:43.827042 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:43.827047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:43.827049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:43.827185 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:43.827544 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:43.827547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:43.827549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:43.827938 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:43.828130 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:43.828562 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 16:40:43.828792 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:40:43.833240 26178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:40:43.840334 26178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:40:43.877112 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.853278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46421 (local address 127.0.0.1:39370)
0926 16:40:43.866262 (+ 12984us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:43.866309 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:43.866492 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:43.867354 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:43.867362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:43.867394 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:43.867817 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.867872 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.871504 (+ 3632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.871510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.873883 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.873900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.876024 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.876031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.876248 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.876315 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 16:40:43.876450 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:40:43.876646 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12835,"mutex_wait_us":95,"spinlock_wait_cycles":7936,"thread_start_us":13379,"threads_started":1}
I20260926 16:40:43.877251 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.853186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:45267 (local address 127.0.0.1:47836)
0926 16:40:43.866254 (+ 13068us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:43.866307 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:43.866492 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:43.868090 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:43.868095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:43.868122 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:43.868559 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.868570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.870868 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.870874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.874140 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.874151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.876345 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.876351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.876538 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.876570 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:40:43.876644 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:40:43.876780 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12766,"mutex_wait_us":588,"spinlock_wait_cycles":2304,"thread_start_us":12777,"threads_started":1}
I20260926 16:40:43.877995 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.854097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.10.110.2:46421)
0926 16:40:43.865892 (+ 11795us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:43.865903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:43.866704 (+ 801us) server_negotiation.cc:407] Connection header received
0926 16:40:43.866821 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:43.866830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:43.867105 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:43.868072 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:43.868318 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.871334 (+ 3016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.874085 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.875789 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.877363 (+ 1574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.877565 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:43.877645 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 16:40:43.877772 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11613,"thread_start_us":11474,"threads_started":1}
I20260926 16:40:43.878062 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.853689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.10.110.1:45267)
0926 16:40:43.867165 (+ 13476us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:43.867173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:43.867200 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:40:43.867289 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:43.867296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:43.867551 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:43.867808 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:43.868760 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.870660 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.874327 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.875792 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.875956 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.876203 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:43.877526 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0926 16:40:43.877712 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13273,"thread_start_us":13096,"threads_started":1}
I20260926 16:40:43.881377 26436 tablet_service.cc:1467] Processing CreateTablet for tablet 784c38d951404fe09aeb3d0e3dcab698 (DEFAULT_TABLE table=test-table [id=c90ff34f005d48cb8c95a261a436a6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:43.881484 26302 tablet_service.cc:1467] Processing CreateTablet for tablet 784c38d951404fe09aeb3d0e3dcab698 (DEFAULT_TABLE table=test-table [id=c90ff34f005d48cb8c95a261a436a6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:43.882766 26436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784c38d951404fe09aeb3d0e3dcab698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:43.882907 26302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784c38d951404fe09aeb3d0e3dcab698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:43.882920 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.867249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38107 (local address 127.0.0.1:51298)
0926 16:40:43.873824 (+ 6575us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:43.873847 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:43.873970 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:43.874707 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:43.874711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:43.874740 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:43.875087 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.875099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.877053 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.877059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.880208 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:43.880226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.882123 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.882130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:43.882294 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.882327 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:40:43.882425 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 16:40:43.882572 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6413,"thread_start_us":6813,"threads_started":1}
I20260926 16:40:43.883464 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:43.867388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.10.110.3:38107)
0926 16:40:43.873354 (+ 5966us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:43.873363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:43.873973 (+ 610us) server_negotiation.cc:407] Connection header received
0926 16:40:43.874122 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:43.874131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:43.874507 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:43.874716 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:43.875275 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.876893 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.880402 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:43.881834 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:43.881982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:43.882205 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:43.882508 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 16:40:43.882677 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5764,"thread_start_us":5584,"threads_started":1}
I20260926 16:40:43.887593 26569 tablet_service.cc:1467] Processing CreateTablet for tablet 784c38d951404fe09aeb3d0e3dcab698 (DEFAULT_TABLE table=test-table [id=c90ff34f005d48cb8c95a261a436a6e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:40:43.889315 26569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784c38d951404fe09aeb3d0e3dcab698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:40:43.895032 26653 tablet_bootstrap.cc:492] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Bootstrap starting.
I20260926 16:40:43.898168 26654 tablet_bootstrap.cc:492] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Bootstrap starting.
I20260926 16:40:43.898674 26653 tablet_bootstrap.cc:654] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:43.900202 26653 log.cc:824] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:43.901787 26654 tablet_bootstrap.cc:654] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:43.903046 26654 log.cc:824] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:43.904291 26656 tablet_bootstrap.cc:492] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Bootstrap starting.
I20260926 16:40:43.905364 26653 tablet_bootstrap.cc:492] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: No bootstrap required, opened a new log
I20260926 16:40:43.905702 26653 ts_tablet_manager.cc:1397] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 16:40:43.907402 26656 tablet_bootstrap.cc:654] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Neither blocks nor log segments found. Creating new log.
I20260926 16:40:43.908743 26656 log.cc:824] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Log is configured to *not* fsync() on all Append() calls
I20260926 16:40:43.910904 26654 tablet_bootstrap.cc:492] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: No bootstrap required, opened a new log
I20260926 16:40:43.911231 26654 ts_tablet_manager.cc:1397] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260926 16:40:43.911371 26656 tablet_bootstrap.cc:492] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: No bootstrap required, opened a new log
I20260926 16:40:43.911620 26656 ts_tablet_manager.cc:1397] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 16:40:43.916186 26653 raft_consensus.cc:348] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:43.916710 26653 raft_consensus.cc:374] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:43.916860 26653 raft_consensus.cc:729] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cf59495051424a8db92217b2ace2fa, State: Initialized, Role: FOLLOWER
I20260926 16:40:43.917986 26653 consensus_queue.cc:260] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:43.920425 26653 ts_tablet_manager.cc:1428] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 16:40:43.920245 26656 raft_consensus.cc:348] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:43.920814 26656 raft_consensus.cc:374] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:43.920926 26656 raft_consensus.cc:729] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d56d0f8839b4147a39b4293c8480238, State: Initialized, Role: FOLLOWER
I20260926 16:40:43.921499 26654 raft_consensus.cc:348] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:43.921597 26656 consensus_queue.cc:260] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:43.921844 26654 raft_consensus.cc:374] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:40:43.921941 26654 raft_consensus.cc:729] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ce0165ab346b68cb4bb49e48bf529, State: Initialized, Role: FOLLOWER
I20260926 16:40:43.922564 26654 consensus_queue.cc:260] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:43.922945 26634 heartbeater.cc:503] Master 127.10.110.62:34537 was elected leader, sending a full tablet report...
I20260926 16:40:43.923585 26656 ts_tablet_manager.cc:1428] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 16:40:43.924643 26501 heartbeater.cc:503] Master 127.10.110.62:34537 was elected leader, sending a full tablet report...
I20260926 16:40:43.925277 26654 ts_tablet_manager.cc:1428] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260926 16:40:43.993422 26502 tablet.cc:2369] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:40:43.999962 26368 tablet.cc:2369] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:40:44.028465 26635 tablet.cc:2369] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:40:44.039251 26659 raft_consensus.cc:482] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:40:44.039559 26659 raft_consensus.cc:504] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:44.041568 26659 leader_election.cc:288] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421), 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107)
I20260926 16:40:44.049222 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.042084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:52099 (local address 127.10.110.2:46421)
0926 16:40:44.042339 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:44.042345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:44.042660 (+ 315us) server_negotiation.cc:407] Connection header received
0926 16:40:44.042878 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:44.042883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:44.042964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:44.043052 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:44.043607 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.044549 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.047021 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.048209 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.048389 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.048492 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:44.048849 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 16:40:44.048960 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:40:44.049388 26662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.041884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46421 (local address 127.10.110.1:52099)
0926 16:40:44.042513 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:44.042529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:44.042723 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:44.043124 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:44.043128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:44.043153 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:44.043452 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:44.043464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.044740 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.044745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:44.046819 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:44.046834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.048480 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.048488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:44.048636 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.048678 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 16:40:44.048820 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:40:44.048971 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":1,"thread_start_us":316,"threads_started":1}
I20260926 16:40:44.049459 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.042840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:54825 (local address 127.10.110.3:38107)
0926 16:40:44.043030 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:44.043036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:44.043350 (+ 314us) server_negotiation.cc:407] Connection header received
0926 16:40:44.043526 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:44.043531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:44.043602 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:44.043691 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:40:44.044199 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.045129 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.047324 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.048532 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.048658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.048708 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:44.049093 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 16:40:44.049247 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:40:44.049654 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.042622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38107 (local address 127.10.110.1:54825)
0926 16:40:44.043248 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 16:40:44.043261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:40:44.043392 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:40:44.043724 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:40:44.043728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:40:44.043751 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:40:44.044062 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:44.044070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.045272 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.045277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:44.047149 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:40:44.047159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.048747 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.048753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:40:44.048922 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.048974 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 16:40:44.049065 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:40:44.049217 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":317,"threads_started":1}
I20260926 16:40:44.050572 26456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c38d951404fe09aeb3d0e3dcab698" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ce0165ab346b68cb4bb49e48bf529" is_pre_election: true
I20260926 16:40:44.050700 26589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c38d951404fe09aeb3d0e3dcab698" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d56d0f8839b4147a39b4293c8480238" is_pre_election: true
I20260926 16:40:44.051108 26456 raft_consensus.cc:2454] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 0.
I20260926 16:40:44.051172 26589 raft_consensus.cc:2454] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 0.
I20260926 16:40:44.051805 26257 leader_election.cc:302] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ce0165ab346b68cb4bb49e48bf529, b3cf59495051424a8db92217b2ace2fa; no voters:
I20260926 16:40:44.052275 26659 raft_consensus.cc:2781] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:40:44.052402 26659 raft_consensus.cc:482] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:40:44.052510 26659 raft_consensus.cc:3037] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:44.054693 26659 raft_consensus.cc:504] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:44.055367 26659 leader_election.cc:288] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 election: Requested vote from peers 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421), 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107)
I20260926 16:40:44.055960 26456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c38d951404fe09aeb3d0e3dcab698" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ce0165ab346b68cb4bb49e48bf529"
I20260926 16:40:44.056056 26589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784c38d951404fe09aeb3d0e3dcab698" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d56d0f8839b4147a39b4293c8480238"
I20260926 16:40:44.056157 26456 raft_consensus.cc:3037] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:44.056284 26589 raft_consensus.cc:3037] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:40:44.058454 26456 raft_consensus.cc:2454] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 1.
I20260926 16:40:44.058526 26589 raft_consensus.cc:2454] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 1.
I20260926 16:40:44.058946 26257 leader_election.cc:302] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ce0165ab346b68cb4bb49e48bf529, b3cf59495051424a8db92217b2ace2fa; no voters:
I20260926 16:40:44.059288 26659 raft_consensus.cc:2781] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:40:44.059818 26659 raft_consensus.cc:686] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 1 LEADER]: Becoming Leader. State: Replica: b3cf59495051424a8db92217b2ace2fa, State: Running, Role: LEADER
I20260926 16:40:44.060413 26659 consensus_queue.cc:237] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } }
I20260926 16:40:44.065052 26178 catalog_manager.cc:5495] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa reported cstate change: term changed from 0 to 1, leader changed from <none> to b3cf59495051424a8db92217b2ace2fa (127.10.110.1). New cstate: current_term: 1 leader_uuid: "b3cf59495051424a8db92217b2ace2fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } health_report { overall_health: UNKNOWN } } }
I20260926 16:40:44.103946 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.098219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.10.110.1:45267)
0926 16:40:44.098395 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:44.098400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:44.098419 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:40:44.098631 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:44.098635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:44.098720 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:44.098865 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:40:44.099163 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.100054 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.101522 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.101963 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.102081 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.102690 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:40:44.103234 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:40:44.103378 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:44.103511 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 16:40:44.103615 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:40:44.114846 26456 raft_consensus.cc:1261] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Refusing update from remote peer b3cf59495051424a8db92217b2ace2fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:44.114965 26589 raft_consensus.cc:1261] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Refusing update from remote peer b3cf59495051424a8db92217b2ace2fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:40:44.116058 26659 consensus_queue.cc:1035] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:44.116456 26664 consensus_queue.cc:1035] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:40:44.142839 26178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:44.143155 26178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:40:44.147219 26178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Sending DeleteTablet for 3 replicas of tablet 784c38d951404fe09aeb3d0e3dcab698
I20260926 16:40:44.148563 26302 tablet_service.cc:1514] Processing DeleteTablet for tablet 784c38d951404fe09aeb3d0e3dcab698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:44 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:47836
I20260926 16:40:44.148839 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26239
I20260926 16:40:44.149051 26436 tablet_service.cc:1514] Processing DeleteTablet for tablet 784c38d951404fe09aeb3d0e3dcab698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:44 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:39370
I20260926 16:40:44.149379 26569 tablet_service.cc:1514] Processing DeleteTablet for tablet 784c38d951404fe09aeb3d0e3dcab698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:40:44 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:51298
I20260926 16:40:44.160247 26677 tablet_replica.cc:330] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: stopping tablet replica
I20260926 16:40:44.160796 26677 raft_consensus.cc:2229] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:40:44.161533 26677 raft_consensus.cc:2258] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:44.163735 26680 tablet_replica.cc:330] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: stopping tablet replica
I20260926 16:40:44.164413 26680 raft_consensus.cc:2229] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:44.165126 26678 tablet_replica.cc:330] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: stopping tablet replica
I20260926 16:40:44.165766 26680 raft_consensus.cc:2258] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:44.166332 26678 raft_consensus.cc:2229] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:40:44.166536 26677 ts_tablet_manager.cc:1905] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:44.168190 26678 raft_consensus.cc:2258] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:40:44.169965 26680 ts_tablet_manager.cc:1905] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:44.172497 26678 ts_tablet_manager.cc:1905] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:40:44.175941 26677 ts_tablet_manager.cc:1918] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:44.176154 26677 log.cc:1196] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/784c38d951404fe09aeb3d0e3dcab698
I20260926 16:40:44.176715 26677 ts_tablet_manager.cc:1939] T 784c38d951404fe09aeb3d0e3dcab698 P b3cf59495051424a8db92217b2ace2fa: Deleting consensus metadata
I20260926 16:40:44.178247 26680 ts_tablet_manager.cc:1918] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:44.178426 26680 log.cc:1196] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/784c38d951404fe09aeb3d0e3dcab698
I20260926 16:40:44.178761 26163 catalog_manager.cc:4841] TS b3cf59495051424a8db92217b2ace2fa (127.10.110.1:45267): tablet 784c38d951404fe09aeb3d0e3dcab698 (table test-table [id=c90ff34f005d48cb8c95a261a436a6e6]) successfully deleted
I20260926 16:40:44.178941 26680 ts_tablet_manager.cc:1939] T 784c38d951404fe09aeb3d0e3dcab698 P 0d56d0f8839b4147a39b4293c8480238: Deleting consensus metadata
I20260926 16:40:44.180855 26164 catalog_manager.cc:4841] TS 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107): tablet 784c38d951404fe09aeb3d0e3dcab698 (table test-table [id=c90ff34f005d48cb8c95a261a436a6e6]) successfully deleted
I20260926 16:40:44.184227 26678 ts_tablet_manager.cc:1918] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:40:44.184458 26678 log.cc:1196] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/784c38d951404fe09aeb3d0e3dcab698
I20260926 16:40:44.185087 26678 ts_tablet_manager.cc:1939] T 784c38d951404fe09aeb3d0e3dcab698 P 363ce0165ab346b68cb4bb49e48bf529: Deleting consensus metadata
Sep 26 16:40:44 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:40:44.187073 26164 catalog_manager.cc:4841] TS 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421): tablet 784c38d951404fe09aeb3d0e3dcab698 (table test-table [id=c90ff34f005d48cb8c95a261a436a6e6]) successfully deleted
I20260926 16:40:44.193215 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.149729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.10.110.1:45267)
0926 16:40:44.149905 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:44.149911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:44.171039 (+ 21128us) server_negotiation.cc:407] Connection header received
0926 16:40:44.173020 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:44.173027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:44.173112 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:44.173179 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:44.178573 (+ 5394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.179774 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.181747 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.182204 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.182330 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.186888 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:44.186909 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:44.186929 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:44.186977 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:44.190052 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:44.190895 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:44.190904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:44.190906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:44.191073 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:44.191477 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:44.191481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:44.191483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:44.191931 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:44.192249 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:44.192617 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 16:40:44.192878 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:40:44.193981 26362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:44.310099 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26239
I20260926 16:40:44.326586 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26373
Sep 26 16:40:44 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:40:44.341784 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.327423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.10.110.2:46421)
0926 16:40:44.327643 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:44.327649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:44.327676 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:40:44.327739 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:44.327742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:44.327802 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:44.327906 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:44.329234 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.330037 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.331513 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.332034 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.332151 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.336215 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:44.336230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:44.336247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:44.336286 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:44.339022 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:44.339640 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:44.339648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:44.339650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:44.339781 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:44.340223 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:44.340228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:44.340230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:44.340646 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:44.340928 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:44.341201 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 16:40:44.341410 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:40:44.342464 26496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:44.443513 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26373
I20260926 16:40:44.457944 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26506
Sep 26 16:40:44 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:40:44.475277 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:40:44.458768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.10.110.3:38107)
0926 16:40:44.458970 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:40:44.458976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:40:44.458998 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:40:44.459094 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:40:44.459099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:40:44.459186 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:40:44.459274 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:40:44.460666 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.461373 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.462769 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:40:44.463173 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:40:44.463289 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:40:44.468843 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:40:44.468861 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:40:44.468879 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:40:44.468964 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:40:44.472004 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:44.472957 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:44.472967 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:44.472969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:44.473143 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:40:44.473605 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:40:44.473609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:40:44.473611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:40:44.474020 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:40:44.474263 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 16:40:44.474707 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 16:40:44.474909 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:40:44.476096 26629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:44.574164 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26506
I20260926 16:40:44.587498 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26146
I20260926 16:40:44.588606 26208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:40:44.699116 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26146
I20260926 16:41:04.717043 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34537
--webserver_interface=127.10.110.62
--webserver_port=40653
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:05.049944 26685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:05.050238 26685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:05.050391 26685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:05.058213 26685 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:41:05.058353 26685 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:41:05.058425 26685 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:41:05.058483 26685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:41:05.058543 26685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:05.058619 26685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:05.058662 26685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:41:05.058714 26685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:41:05.068956 26685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=40653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:05.070286 26685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:05.071607 26685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:05.093537 26694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:05.093515 26691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:05.093608 26692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:05.094201 26685 server_base.cc:1034] running on GCE node
I20260926 16:41:05.095090 26685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:05.097298 26685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:05.098548 26685 hybrid_clock.cc:648] HybridClock initialized: now 1790440865098475 us; error 50 us; skew 500 ppm
I20260926 16:41:05.098968 26685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:05 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440865, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:05.102574 26685 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:41:05.104301 26685 webserver.cc:460] Webserver started at http://127.10.110.62:40653/ using document root <none> and password file <none>
I20260926 16:41:05.105094 26685 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:05.105221 26685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:05.111541 26685 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 16:41:05.115007 26701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:05.116649 26685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:05.116845 26685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "b054271b34104531b2d1bf3f4cb75e73"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "cb6451e9d9baf62d146bfd1bdea4b250"
server_key_iv: "2372400fdd724ed1a92b552278a76ae1"
server_key_version: "encryptionkey@0"
I20260926 16:41:05.117931 26685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:05.146370 26685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:05.150913 26685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:05.151197 26685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:05.169718 26685 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:34537
I20260926 16:41:05.169754 26752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:34537 every 8 connection(s)
I20260926 16:41:05.171216 26685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:41:05.178411 26753 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:41:05.181238 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26685
I20260926 16:41:05.182241 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:45267
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:34537
--webserver_port=34867
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:41:05.182494 26753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap starting.
I20260926 16:41:05.204727 26753 log.cc:824] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:05.218232 26753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:41:05.218833 26753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap complete.
I20260926 16:41:05.228495 26753 raft_consensus.cc:348] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:05.229408 26753 raft_consensus.cc:729] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b054271b34104531b2d1bf3f4cb75e73, State: Initialized, Role: FOLLOWER
I20260926 16:41:05.230052 26753 consensus_queue.cc:260] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:05.230350 26753 raft_consensus.cc:388] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:41:05.230486 26753 raft_consensus.cc:482] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:41:05.230656 26753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:41:05.234714 26753 raft_consensus.cc:504] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:05.235283 26753 leader_election.cc:302] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b054271b34104531b2d1bf3f4cb75e73; no voters:
I20260926 16:41:05.235947 26753 leader_election.cc:288] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:41:05.236089 26757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:41:05.237222 26757 raft_consensus.cc:686] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 LEADER]: Becoming Leader. State: Replica: b054271b34104531b2d1bf3f4cb75e73, State: Running, Role: LEADER
I20260926 16:41:05.237897 26757 consensus_queue.cc:237] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:05.238368 26753 sys_catalog.cc:564] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:41:05.240520 26759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b054271b34104531b2d1bf3f4cb75e73. Latest consensus state: current_term: 2 leader_uuid: "b054271b34104531b2d1bf3f4cb75e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } } }
I20260926 16:41:05.240933 26759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:05.240752 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b054271b34104531b2d1bf3f4cb75e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } } }
I20260926 16:41:05.241133 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:05.253100 26764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:41:05.262312 26764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:41:05.263871 26764 catalog_manager.cc:1259] Loaded cluster ID: f03eda8b4da048fb9a0ce1006cfcbd51
I20260926 16:41:05.264066 26764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:41:05.268278 26764 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:41:05.270694 26764 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Loaded TSK: 0
I20260926 16:41:05.300495 26764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Generated new TSK 1
I20260926 16:41:05.300930 26764 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:41:05.546217 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:05.546538 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:05.546681 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:05.554622 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:05.554782 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:05.554936 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:41:05.565364 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:45267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=34867
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:05.566692 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:05.568084 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:05.582492 26781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:05.582648 26780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:05.582610 26783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:05.583348 26755 server_base.cc:1034] running on GCE node
I20260926 16:41:05.584106 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:05.585217 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:05.586485 26755 hybrid_clock.cc:648] HybridClock initialized: now 1790440865586410 us; error 57 us; skew 500 ppm
I20260926 16:41:05.586898 26755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:05 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440865, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:05.590922 26755 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:41:05.592700 26755 webserver.cc:460] Webserver started at http://127.10.110.1:34867/ using document root <none> and password file <none>
I20260926 16:41:05.593420 26755 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:05.593568 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:05.599545 26755 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:41:05.602659 26790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:05.604274 26755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:41:05.604442 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "b3cf59495051424a8db92217b2ace2fa"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "674c6f5cb1e7ed0e1e278a8a4ce892a4"
server_key_iv: "3ab3491d5100aec63fd3594be55b2425"
server_key_version: "encryptionkey@0"
I20260926 16:41:05.605592 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:05.637434 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:05.642303 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:05.642587 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:05.643621 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:05.645059 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:05.645157 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:05.645263 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:05.645334 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:05.676396 26755 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:45267
I20260926 16:41:05.676469 26902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:45267 every 8 connection(s)
I20260926 16:41:05.677806 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:41:05.679034 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26755
I20260926 16:41:05.679646 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:46421
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:34537
--webserver_port=35857
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:41:05 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440865, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:05.705125 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:05.681175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:38005)
0926 16:41:05.681942 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:05.681985 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:05.683717 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:05.685439 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:05.685450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:05.686106 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:05.686880 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:05.686907 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:05.689449 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:05.689460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:05.690834 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:05.690861 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:05.692343 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:05.692349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:05.692563 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:05.693253 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:05.693277 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:05.695676 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:05.700448 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:05.700456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:05.700473 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:05.700996 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:05.701556 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:05.701560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:05.701562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:05.701769 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:05.702832 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:05.702840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:05.703220 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 16:41:05.703482 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 16:41:05.703994 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":289,"threads_started":1}
I20260926 16:41:05.705637 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:05.680765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:38005 (local address 127.10.110.62:34537)
0926 16:41:05.681364 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:05.681377 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:05.682134 (+ 757us) server_negotiation.cc:407] Connection header received
0926 16:41:05.684399 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:05.684414 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:05.684834 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:05.685211 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:05.687229 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:05.689217 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:05.691012 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:05.692083 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:05.692275 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:05.696066 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:05.696079 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:05.696101 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:05.696160 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:05.700236 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:05.701162 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:05.701173 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:05.701181 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:05.701435 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:05.701881 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:05.701885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:05.701888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:05.702631 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:05.702867 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:05.703778 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 16:41:05.704302 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260926 16:41:05.708549 26903 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34537
I20260926 16:41:05.708887 26903 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:05.709692 26903 heartbeater.cc:511] Master 127.10.110.62:34537 requested a full tablet report, sending...
I20260926 16:41:05.712045 26718 ts_manager.cc:194] Registered new tserver with Master: b3cf59495051424a8db92217b2ace2fa (127.10.110.1:45267)
I20260926 16:41:05.713503 26718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:38005
W20260926 16:41:06.014652 26907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:06.014984 26907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:06.015130 26907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:06.022509 26907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:06.022666 26907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:06.022854 26907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:41:06.032939 26907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:46421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=35857
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:06.034332 26907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:06.035651 26907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:06.049101 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:06.049088 26915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:06.049077 26913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:06.050014 26907 server_base.cc:1034] running on GCE node
I20260926 16:41:06.050729 26907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:06.051730 26907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:06.052973 26907 hybrid_clock.cc:648] HybridClock initialized: now 1790440866052907 us; error 58 us; skew 500 ppm
I20260926 16:41:06.053385 26907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:06 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440866, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:06.056535 26907 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:41:06.058005 26907 webserver.cc:460] Webserver started at http://127.10.110.2:35857/ using document root <none> and password file <none>
I20260926 16:41:06.058686 26907 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:06.058816 26907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:06.064276 26907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:06.067104 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:06.068611 26907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:06.068768 26907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "363ce0165ab346b68cb4bb49e48bf529"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "1a85bcc21ee26664e9bdadcaa7659653"
server_key_iv: "30304473f96b3da3fade3ee0c690bfc1"
server_key_version: "encryptionkey@0"
I20260926 16:41:06.069782 26907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:06.083921 26907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:06.087987 26907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:06.088248 26907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:06.089265 26907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:06.090608 26907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:06.090708 26907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:06.090811 26907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:06.090878 26907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:06.120221 26907 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:46421
I20260926 16:41:06.120292 27034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:46421 every 8 connection(s)
I20260926 16:41:06.121730 26907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:41:06.125254 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 26907
I20260926 16:41:06.126142 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:38107
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:34537
--webserver_port=40925
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:41:06 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440866, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:06.145754 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.124630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:48377 (local address 127.10.110.62:34537)
0926 16:41:06.124822 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.124827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.126673 (+ 1846us) server_negotiation.cc:407] Connection header received
0926 16:41:06.128878 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.128885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.128982 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.129071 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:06.130759 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.131476 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.133824 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.134632 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.134801 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.139243 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:06.139249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:06.139253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:06.139294 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:06.142098 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.143146 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.143151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.143153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.143298 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.143822 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.143855 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.143859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.144178 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:06.144325 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.145143 (+ 818us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.145401 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:41:06.146788 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.125471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:48377)
0926 16:41:06.126474 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:06.126536 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:06.128545 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:06.129359 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:06.129369 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:06.129936 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:06.130559 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.130579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.131641 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.131646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.133556 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.133574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.134936 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.134945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.135246 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.136039 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:06.136069 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:06.138883 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:06.142299 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:06.142310 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:06.142333 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:06.142992 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:06.143451 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:06.143455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:06.143458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:06.143678 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:06.144321 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:06.144329 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:06.144740 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 16:41:06.144996 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 16:41:06.145588 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":366,"threads_started":1}
I20260926 16:41:06.148447 27035 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34537
I20260926 16:41:06.148830 27035 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:06.149799 27035 heartbeater.cc:511] Master 127.10.110.62:34537 requested a full tablet report, sending...
I20260926 16:41:06.151504 26718 ts_manager.cc:194] Registered new tserver with Master: 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421)
I20260926 16:41:06.152652 26718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:48377
W20260926 16:41:06.460208 27038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:06.460507 27038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:06.460599 27038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:06.468011 27038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:06.468137 27038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:06.468257 27038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:41:06.478278 27038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:38107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=40925
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:06.479535 27038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:06.480917 27038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:06.494050 27043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:06.494010 27044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:06.494069 27046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:06.494815 27038 server_base.cc:1034] running on GCE node
I20260926 16:41:06.495441 27038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:06.496484 27038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:06.497694 27038 hybrid_clock.cc:648] HybridClock initialized: now 1790440866497636 us; error 47 us; skew 500 ppm
I20260926 16:41:06.498013 27038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:06 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440866, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:06.501286 27038 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:41:06.502902 27038 webserver.cc:460] Webserver started at http://127.10.110.3:40925/ using document root <none> and password file <none>
I20260926 16:41:06.503625 27038 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:06.503764 27038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:06.509594 27038 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:41:06.512691 27053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:06.514441 27038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:41:06.514652 27038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "0d56d0f8839b4147a39b4293c8480238"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "5a38c07d7d37717e4d19f32f2fc4dd4c"
server_key_iv: "d6140bed81b4c3d60c00c86044717ed4"
server_key_version: "encryptionkey@0"
I20260926 16:41:06.515663 27038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:06.542069 27038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:06.546372 27038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:06.546624 27038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:06.547595 27038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:06.549081 27038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:06.549175 27038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:06.549295 27038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:06.549360 27038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:06.581101 27038 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38107
I20260926 16:41:06.581177 27165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38107 every 8 connection(s)
I20260926 16:41:06.582587 27038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:41:06.592319 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27038
Sep 26 16:41:06 dist-test-slave-whjp krb5kdc[26133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440866, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:06.604421 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.585140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:53407 (local address 127.10.110.62:34537)
0926 16:41:06.585338 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.585343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.586539 (+ 1196us) server_negotiation.cc:407] Connection header received
0926 16:41:06.588475 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.588481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.588569 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.588655 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:06.590305 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.591085 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.593170 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.593855 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.594004 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.597773 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:06.597781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:06.597786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:06.597829 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:06.600692 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.601683 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.601687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.601689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.601804 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.602363 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.602369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.602372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.602741 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:06.602890 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.603822 (+ 932us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.604090 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:41:06.605472 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.585646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:53407)
0926 16:41:06.586367 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:06.586411 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:06.588195 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:06.588916 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:06.588925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:06.589460 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:06.590099 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.590119 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.591253 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.591257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.593009 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.593024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.594334 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.594339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.594555 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.595265 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:06.595290 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:06.597547 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:06.600860 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:06.600873 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:06.600893 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:06.601563 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:06.601908 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:06.601913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:06.601917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:06.602209 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:06.602886 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:06.602893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:06.603367 (+ 474us) client_negotiation.cc:769] Sending connection context
0926 16:41:06.603677 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 16:41:06.604173 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":265,"threads_started":1}
I20260926 16:41:06.607118 27166 heartbeater.cc:347] Connected to a master server at 127.10.110.62:34537
I20260926 16:41:06.607478 27166 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:06.608443 27166 heartbeater.cc:511] Master 127.10.110.62:34537 requested a full tablet report, sending...
I20260926 16:41:06.608471 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.593369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.10.110.62:34537)
0926 16:41:06.594253 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.594259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.594275 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:41:06.594317 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.594320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.594386 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.594473 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:06.596184 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.596997 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.598624 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.599080 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.599205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.602472 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:06.602484 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:06.602489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:06.602524 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:06.605514 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.606381 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.606385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.606388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.606515 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.606913 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.606917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.606919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.607284 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:06.607547 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.607866 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.608091 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":598,"threads_started":1}
I20260926 16:41:06.609917 26718 ts_manager.cc:194] Registered new tserver with Master: 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107)
I20260926 16:41:06.611079 26718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:53407
I20260926 16:41:06.611279 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:41:06.617588 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.612206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.10.110.62:34537)
0926 16:41:06.612398 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.612404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.612766 (+ 362us) server_negotiation.cc:407] Connection header received
0926 16:41:06.612953 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.612958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.613131 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.613337 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:41:06.613676 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.614453 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.615961 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.616376 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.616470 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.616883 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:41:06.617081 (+ 198us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 16:41:06.617358 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43906: token expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 16:41:06.617730 27170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43906: token expired
W20260926 16:41:06.618273 26641 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.110.62:34537)
I20260926 16:41:06.631358 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.618749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.10.110.62:34537)
0926 16:41:06.618898 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.618902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.618916 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:41:06.618975 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.618979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.619032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.619162 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:06.620625 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.621365 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.622736 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.623223 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.623335 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.625858 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:06.625863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:06.625866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:06.625900 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:06.628652 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.629396 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.629401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.629403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.629547 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:06.630045 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:06.630049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:06.630051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:06.630420 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:06.630653 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.630920 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.631108 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:41:06.634959 26718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:41:06.642474 26718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:41:06.669544 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.655675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.10.110.1:45267)
0926 16:41:06.657200 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.657212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.658664 (+ 1452us) server_negotiation.cc:407] Connection header received
0926 16:41:06.659002 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.659014 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.659344 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.659533 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:06.660649 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.662500 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.666301 (+ 3801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.667882 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.667996 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.668220 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.668991 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.669192 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":1145,"threads_started":1}
I20260926 16:41:06.670859 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.654966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:45267 (local address 127.0.0.1:60638)
0926 16:41:06.658521 (+ 3555us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:06.658544 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:06.658793 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:06.659933 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:06.659942 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:06.659978 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:06.660440 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.660453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.662729 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.662735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.665547 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.665574 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.668448 (+ 2874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.668455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.668682 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.668751 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 16:41:06.670053 (+ 1302us) client_negotiation.cc:240] Negotiation successful
0926 16:41:06.670374 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3413,"mutex_wait_us":37,"thread_start_us":3237,"threads_started":1}
I20260926 16:41:06.674780 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.654841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38107 (local address 127.0.0.1:50680)
0926 16:41:06.655730 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:06.655774 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:06.656008 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:06.667266 (+ 11258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:06.667273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:06.667307 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:06.667694 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.667708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.669713 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.669719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.671869 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.671885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.673986 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.673993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.674181 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.674217 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:41:06.674305 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:41:06.674445 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":392,"threads_started":1}
I20260926 16:41:06.675536 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.655046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.10.110.3:38107)
0926 16:41:06.666272 (+ 11226us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.666282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.666320 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:41:06.666428 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.666437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.666740 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.666962 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:06.667954 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.669511 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.672059 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.673456 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.673583 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.673821 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.675043 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.675221 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11034,"thread_start_us":10871,"threads_started":1}
I20260926 16:41:06.675523 26838 tablet_service.cc:1467] Processing CreateTablet for tablet 0465b75f5ce1411bb018140e7f4637aa (DEFAULT_TABLE table=test-table [id=6a72b826a030439bbd0580f5802d9b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:06.677264 26838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0465b75f5ce1411bb018140e7f4637aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:06.680109 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.658317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46421 (local address 127.0.0.1:44106)
0926 16:41:06.671278 (+ 12961us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:06.671298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:06.671418 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:06.672574 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:06.672579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:06.672607 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:06.672803 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.672814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.675152 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.675158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.677075 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:06.677103 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.679265 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.679272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:06.679468 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.679507 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:41:06.679584 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 16:41:06.679773 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12856,"thread_start_us":15366,"threads_started":1}
I20260926 16:41:06.680193 27101 tablet_service.cc:1467] Processing CreateTablet for tablet 0465b75f5ce1411bb018140e7f4637aa (DEFAULT_TABLE table=test-table [id=6a72b826a030439bbd0580f5802d9b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:06.681190 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:06.658779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.10.110.2:46421)
0926 16:41:06.671927 (+ 13148us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:06.671936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:06.671967 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:41:06.672076 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:06.672085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:06.672353 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:06.672985 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:06.673192 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.674714 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.677310 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:06.679013 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:06.680467 (+ 1454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:06.680657 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:06.680759 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 16:41:06.680922 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12951,"threads_started":1}
I20260926 16:41:06.681751 27101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0465b75f5ce1411bb018140e7f4637aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:06.686159 26970 tablet_service.cc:1467] Processing CreateTablet for tablet 0465b75f5ce1411bb018140e7f4637aa (DEFAULT_TABLE table=test-table [id=6a72b826a030439bbd0580f5802d9b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:06.688283 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0465b75f5ce1411bb018140e7f4637aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:06.694763 27182 tablet_bootstrap.cc:492] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Bootstrap starting.
I20260926 16:41:06.698863 27182 tablet_bootstrap.cc:654] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:06.698850 27183 tablet_bootstrap.cc:492] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Bootstrap starting.
I20260926 16:41:06.700476 27182 log.cc:824] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:06.702401 27183 tablet_bootstrap.cc:654] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:06.703449 27185 tablet_bootstrap.cc:492] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Bootstrap starting.
I20260926 16:41:06.704695 27182 tablet_bootstrap.cc:492] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: No bootstrap required, opened a new log
I20260926 16:41:06.704846 27183 log.cc:824] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:06.704996 27182 ts_tablet_manager.cc:1397] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:41:06.706812 27185 tablet_bootstrap.cc:654] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:06.707824 27185 log.cc:824] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:06.708163 27183 tablet_bootstrap.cc:492] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: No bootstrap required, opened a new log
I20260926 16:41:06.708447 27183 ts_tablet_manager.cc:1397] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:41:06.710727 27185 tablet_bootstrap.cc:492] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: No bootstrap required, opened a new log
I20260926 16:41:06.711005 27185 ts_tablet_manager.cc:1397] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 16:41:06.715725 27182 raft_consensus.cc:348] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:06.716284 27182 raft_consensus.cc:374] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:06.716387 27182 raft_consensus.cc:729] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cf59495051424a8db92217b2ace2fa, State: Initialized, Role: FOLLOWER
I20260926 16:41:06.717056 26903 heartbeater.cc:503] Master 127.10.110.62:34537 was elected leader, sending a full tablet report...
I20260926 16:41:06.717059 27182 consensus_queue.cc:260] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:06.719151 27185 raft_consensus.cc:348] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:06.719589 27185 raft_consensus.cc:374] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:06.719693 27185 raft_consensus.cc:729] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ce0165ab346b68cb4bb49e48bf529, State: Initialized, Role: FOLLOWER
I20260926 16:41:06.719672 27183 raft_consensus.cc:348] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:06.720214 27183 raft_consensus.cc:374] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:06.720302 27183 raft_consensus.cc:729] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d56d0f8839b4147a39b4293c8480238, State: Initialized, Role: FOLLOWER
I20260926 16:41:06.720397 27185 consensus_queue.cc:260] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:06.720976 27183 consensus_queue.cc:260] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:06.722096 27035 heartbeater.cc:503] Master 127.10.110.62:34537 was elected leader, sending a full tablet report...
I20260926 16:41:06.722412 27185 ts_tablet_manager.cc:1428] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 16:41:06.723999 27166 heartbeater.cc:503] Master 127.10.110.62:34537 was elected leader, sending a full tablet report...
I20260926 16:41:06.725219 27182 ts_tablet_manager.cc:1428] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
I20260926 16:41:06.726349 27183 ts_tablet_manager.cc:1428] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
W20260926 16:41:06.834153 27167 tablet.cc:2369] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:41:06.873593 27036 tablet.cc:2369] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:41:06.930823 26904 tablet.cc:2369] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:07.018671 27188 raft_consensus.cc:482] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:41:07.018982 27188 raft_consensus.cc:504] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:07.020815 27188 leader_election.cc:288] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107), 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421)
I20260926 16:41:07.022744 27189 raft_consensus.cc:482] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:41:07.022998 27189 raft_consensus.cc:504] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:07.024871 27189 leader_election.cc:288] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107), b3cf59495051424a8db92217b2ace2fa (127.10.110.1:45267)
I20260926 16:41:07.029768 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.021275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:53281 (local address 127.10.110.3:38107)
0926 16:41:07.021537 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.021544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.021892 (+ 348us) server_negotiation.cc:407] Connection header received
0926 16:41:07.022107 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.022131 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.022211 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.022304 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:07.023069 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.024092 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.026595 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.027861 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.027996 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.028083 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.029320 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.029461 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 16:41:07.030467 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.021169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38107 (local address 127.10.110.1:53281)
0926 16:41:07.021783 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:07.021799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:07.021951 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:07.022492 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:07.022496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:07.022551 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:07.022928 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.022941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.024232 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.024237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.026403 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.026419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.028657 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.028663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.029150 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.029179 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:41:07.029929 (+ 750us) client_negotiation.cc:240] Negotiation successful
0926 16:41:07.030074 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":290,"thread_start_us":338,"threads_started":1}
I20260926 16:41:07.031091 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.022299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:58741 (local address 127.10.110.2:46421)
0926 16:41:07.022512 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.022518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.022949 (+ 431us) server_negotiation.cc:407] Connection header received
0926 16:41:07.023184 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.023189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.023249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.023319 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:07.023814 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.024764 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.027094 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.028392 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.028504 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.028569 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.030691 (+ 2122us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.030849 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:41:07.031729 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.021964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:46421 (local address 127.10.110.1:58741)
0926 16:41:07.022846 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:07.022864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:07.023041 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:07.023348 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:07.023352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:07.023376 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:07.023659 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.023668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.024926 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.024930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.026917 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.026930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.028757 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.028762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.028960 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.028993 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:41:07.031259 (+ 2266us) client_negotiation.cc:240] Negotiation successful
0926 16:41:07.031397 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":1,"thread_start_us":348,"threads_started":1}
I20260926 16:41:07.032166 27121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b75f5ce1411bb018140e7f4637aa" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d56d0f8839b4147a39b4293c8480238" is_pre_election: true
I20260926 16:41:07.032775 27121 raft_consensus.cc:2454] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 0.
I20260926 16:41:07.033558 26793 leader_election.cc:302] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d56d0f8839b4147a39b4293c8480238, b3cf59495051424a8db92217b2ace2fa; no voters:
I20260926 16:41:07.033977 27188 raft_consensus.cc:2781] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:41:07.034129 27188 raft_consensus.cc:482] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:41:07.034230 27188 raft_consensus.cc:3037] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:07.034380 26990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b75f5ce1411bb018140e7f4637aa" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ce0165ab346b68cb4bb49e48bf529" is_pre_election: true
I20260926 16:41:07.035100 26990 raft_consensus.cc:2454] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 0.
I20260926 16:41:07.036872 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.025114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:38107 (local address 127.10.110.2:52397)
0926 16:41:07.029271 (+ 4157us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:07.029290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:07.029397 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:07.030464 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:07.030472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:07.030499 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:07.030895 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.030907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.032564 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.032571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.034592 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.034606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.036170 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.036177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.036333 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.036368 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:41:07.036458 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 16:41:07.036569 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4010,"mutex_wait_us":66,"thread_start_us":5159,"threads_started":1}
I20260926 16:41:07.037355 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.026239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:37285 (local address 127.10.110.1:45267)
0926 16:41:07.026436 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.026443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.030721 (+ 4278us) server_negotiation.cc:407] Connection header received
0926 16:41:07.030938 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.030944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.031012 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.031099 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:07.032010 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.032938 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.035240 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.036232 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.036373 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.036441 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.036971 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.037106 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:41:07.037369 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.025781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:45267 (local address 127.10.110.2:37285)
0926 16:41:07.030615 (+ 4834us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:07.030635 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:07.030809 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:07.031356 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:07.031360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:07.031386 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:07.031705 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.031717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.033109 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.033114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.035062 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:07.035082 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.036492 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.036498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:07.036737 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.036774 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:41:07.036876 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:41:07.037009 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4710,"mutex_wait_us":1,"thread_start_us":3159,"threads_started":1}
I20260926 16:41:07.037670 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.025297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:52397 (local address 127.10.110.3:38107)
0926 16:41:07.029929 (+ 4632us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.029933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.029954 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:41:07.030008 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.030013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.030095 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.030186 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:07.031029 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.032416 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.034769 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.035944 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.036075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.036128 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.037229 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.037362 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4489,"thread_start_us":4246,"threads_started":1}
I20260926 16:41:07.038213 26858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b75f5ce1411bb018140e7f4637aa" candidate_uuid: "363ce0165ab346b68cb4bb49e48bf529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cf59495051424a8db92217b2ace2fa" is_pre_election: true
I20260926 16:41:07.038566 27121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b75f5ce1411bb018140e7f4637aa" candidate_uuid: "363ce0165ab346b68cb4bb49e48bf529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d56d0f8839b4147a39b4293c8480238" is_pre_election: true
I20260926 16:41:07.038810 27121 raft_consensus.cc:2454] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363ce0165ab346b68cb4bb49e48bf529 in term 0.
I20260926 16:41:07.039142 27188 raft_consensus.cc:504] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:07.039218 26925 leader_election.cc:302] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d56d0f8839b4147a39b4293c8480238, 363ce0165ab346b68cb4bb49e48bf529; no voters:
I20260926 16:41:07.039588 26858 raft_consensus.cc:2379] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 363ce0165ab346b68cb4bb49e48bf529 in current term 1: Already voted for candidate b3cf59495051424a8db92217b2ace2fa in this term.
I20260926 16:41:07.039889 27188 leader_election.cc:288] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 election: Requested vote from peers 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107), 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421)
I20260926 16:41:07.040408 27121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b75f5ce1411bb018140e7f4637aa" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d56d0f8839b4147a39b4293c8480238"
I20260926 16:41:07.040464 26990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0465b75f5ce1411bb018140e7f4637aa" candidate_uuid: "b3cf59495051424a8db92217b2ace2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ce0165ab346b68cb4bb49e48bf529"
I20260926 16:41:07.040593 27121 raft_consensus.cc:3037] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:07.040670 26990 raft_consensus.cc:3037] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:07.042964 26990 raft_consensus.cc:2454] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 1.
I20260926 16:41:07.042980 27121 raft_consensus.cc:2454] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3cf59495051424a8db92217b2ace2fa in term 1.
I20260926 16:41:07.043125 27189 raft_consensus.cc:2741] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 16:41:07.043406 26793 leader_election.cc:302] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ce0165ab346b68cb4bb49e48bf529, b3cf59495051424a8db92217b2ace2fa; no voters:
I20260926 16:41:07.043699 27188 raft_consensus.cc:2781] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:07.044250 27188 raft_consensus.cc:686] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 1 LEADER]: Becoming Leader. State: Replica: b3cf59495051424a8db92217b2ace2fa, State: Running, Role: LEADER
I20260926 16:41:07.044740 27188 consensus_queue.cc:237] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } }
I20260926 16:41:07.049337 26717 catalog_manager.cc:5495] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa reported cstate change: term changed from 0 to 1, leader changed from <none> to b3cf59495051424a8db92217b2ace2fa (127.10.110.1). New cstate: current_term: 1 leader_uuid: "b3cf59495051424a8db92217b2ace2fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3cf59495051424a8db92217b2ace2fa" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 45267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 } health_report { overall_health: UNKNOWN } } }
I20260926 16:41:07.149214 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.143524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.10.110.1:45267)
0926 16:41:07.143688 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.143693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.144014 (+ 321us) server_negotiation.cc:407] Connection header received
0926 16:41:07.144242 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.144247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.144338 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.144479 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:41:07.144843 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.145582 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.147028 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.147458 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.147557 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.148157 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:41:07.148675 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:41:07.148811 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.148898 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.148975 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:41:07.157877 27121 raft_consensus.cc:1261] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Refusing update from remote peer b3cf59495051424a8db92217b2ace2fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:07.157894 26990 raft_consensus.cc:1261] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Refusing update from remote peer b3cf59495051424a8db92217b2ace2fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:07.158828 27196 consensus_queue.cc:1035] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d56d0f8839b4147a39b4293c8480238" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:07.159224 27188 consensus_queue.cc:1035] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ce0165ab346b68cb4bb49e48bf529" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:07.194926 26717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:07.195216 26717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:07.198580 26717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Sending DeleteTablet for 3 replicas of tablet 0465b75f5ce1411bb018140e7f4637aa
I20260926 16:41:07.199971 26838 tablet_service.cc:1514] Processing DeleteTablet for tablet 0465b75f5ce1411bb018140e7f4637aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:07 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:60638
I20260926 16:41:07.199993 27101 tablet_service.cc:1514] Processing DeleteTablet for tablet 0465b75f5ce1411bb018140e7f4637aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:07 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:50680
I20260926 16:41:07.200187 26970 tablet_service.cc:1514] Processing DeleteTablet for tablet 0465b75f5ce1411bb018140e7f4637aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:07 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:44106
I20260926 16:41:07.201310 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26755
I20260926 16:41:07.210757 27211 tablet_replica.cc:330] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: stopping tablet replica
I20260926 16:41:07.211463 27211 raft_consensus.cc:2229] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:07.212190 27211 raft_consensus.cc:2258] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:07.213053 27209 tablet_replica.cc:330] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: stopping tablet replica
I20260926 16:41:07.215255 27210 tablet_replica.cc:330] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: stopping tablet replica
I20260926 16:41:07.215392 27209 raft_consensus.cc:2229] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:07.215988 27210 raft_consensus.cc:2229] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:41:07.216043 27209 raft_consensus.cc:2258] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:07.216228 27211 ts_tablet_manager.cc:1905] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:07.218272 27209 ts_tablet_manager.cc:1905] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:07.219329 27210 raft_consensus.cc:2258] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:07.225911 27209 ts_tablet_manager.cc:1918] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:07.222956 27210 ts_tablet_manager.cc:1905] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:07.226163 27209 log.cc:1196] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/0465b75f5ce1411bb018140e7f4637aa
I20260926 16:41:07.226748 27209 ts_tablet_manager.cc:1939] T 0465b75f5ce1411bb018140e7f4637aa P 0d56d0f8839b4147a39b4293c8480238: Deleting consensus metadata
I20260926 16:41:07.228874 26704 catalog_manager.cc:4841] TS 0d56d0f8839b4147a39b4293c8480238 (127.10.110.3:38107): tablet 0465b75f5ce1411bb018140e7f4637aa (table test-table [id=6a72b826a030439bbd0580f5802d9b61]) successfully deleted
I20260926 16:41:07.229846 27211 ts_tablet_manager.cc:1918] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:07.230021 27211 log.cc:1196] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/0465b75f5ce1411bb018140e7f4637aa
I20260926 16:41:07.230549 27211 ts_tablet_manager.cc:1939] T 0465b75f5ce1411bb018140e7f4637aa P 363ce0165ab346b68cb4bb49e48bf529: Deleting consensus metadata
I20260926 16:41:07.232818 26704 catalog_manager.cc:4841] TS 363ce0165ab346b68cb4bb49e48bf529 (127.10.110.2:46421): tablet 0465b75f5ce1411bb018140e7f4637aa (table test-table [id=6a72b826a030439bbd0580f5802d9b61]) successfully deleted
I20260926 16:41:07.233935 27210 ts_tablet_manager.cc:1918] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:07.234105 27210 log.cc:1196] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/0465b75f5ce1411bb018140e7f4637aa
I20260926 16:41:07.234583 27210 ts_tablet_manager.cc:1939] T 0465b75f5ce1411bb018140e7f4637aa P b3cf59495051424a8db92217b2ace2fa: Deleting consensus metadata
I20260926 16:41:07.236127 26703 catalog_manager.cc:4841] TS b3cf59495051424a8db92217b2ace2fa (127.10.110.1:45267): tablet 0465b75f5ce1411bb018140e7f4637aa (table test-table [id=6a72b826a030439bbd0580f5802d9b61]) successfully deleted
I20260926 16:41:07.244000 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.202185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.10.110.1:45267)
0926 16:41:07.202367 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.202374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.219789 (+ 17415us) server_negotiation.cc:407] Connection header received
0926 16:41:07.220322 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.220328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.220418 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.220490 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:07.222731 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.223901 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.226236 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.226888 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.227001 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.237169 (+ 10168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:07.237184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:07.237199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:07.237250 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:07.240421 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:07.241448 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:07.241457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:07.241460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:07.241695 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:07.242207 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:07.242212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:07.242215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:07.242745 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:07.242992 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.243392 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.243635 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:41:07.244711 26898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:07.347470 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26755
I20260926 16:41:07.362552 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26907
I20260926 16:41:07.377480 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.363436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.10.110.2:46421)
0926 16:41:07.363703 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.363710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.363738 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:41:07.363801 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.363805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.363903 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.364012 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:07.365315 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.366028 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.367357 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.367789 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.367969 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.370638 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:07.370653 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:07.370669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:07.370707 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:07.373852 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:07.374960 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:07.374969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:07.374972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:07.375270 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:07.375763 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:07.375767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:07.375769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:07.376348 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:07.376593 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.376936 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.377131 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:41:07.378129 27030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:07.468310 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26907
I20260926 16:41:07.482631 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 27038
I20260926 16:41:07.497013 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:07.483477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.10.110.3:38107)
0926 16:41:07.483640 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:07.483645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:07.483664 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:41:07.483725 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:07.483729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:07.483793 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:07.483912 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:07.485266 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.485977 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.487433 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:07.487981 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:07.488105 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:07.490839 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:07.490855 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:07.490873 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:07.490917 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:07.494075 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:07.494895 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:07.494904 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:07.494906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:07.495060 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:07.495492 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:07.495496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:07.495498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:07.495946 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:07.496196 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:07.496529 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 16:41:07.496770 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:41:07.497673 27161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:07.587729 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 27038
I20260926 16:41:07.604748 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 26685
I20260926 16:41:07.605899 26748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:07.721247 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 26685
I20260926 16:41:27.739317 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34537
--webserver_interface=127.10.110.62
--webserver_port=40653
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:28.098445 27217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:28.098783 27217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:28.098901 27217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:28.106701 27217 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:41:28.106827 27217 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:41:28.106865 27217 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:41:28.106921 27217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:41:28.106997 27217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:28.107081 27217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:28.107141 27217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:41:28.107213 27217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:41:28.118150 27217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:34537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=40653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:28.119588 27217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:28.121076 27217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:28.134853 27222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:28.134780 27217 server_base.cc:1034] running on GCE node
W20260926 16:41:28.134821 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:28.134780 27225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:28.136145 27217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:28.138062 27217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:28.139366 27217 hybrid_clock.cc:648] HybridClock initialized: now 1790440888139297 us; error 42 us; skew 500 ppm
I20260926 16:41:28.139778 27217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:28 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:28.149794 27217 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:41:28.151587 27217 webserver.cc:460] Webserver started at http://127.10.110.62:40653/ using document root <none> and password file <none>
I20260926 16:41:28.152423 27217 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:28.152535 27217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:28.158613 27217 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:41:28.162096 27232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:28.163802 27217 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:41:28.164060 27217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "b054271b34104531b2d1bf3f4cb75e73"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "cb6451e9d9baf62d146bfd1bdea4b250"
server_key_iv: "2372400fdd724ed1a92b552278a76ae1"
server_key_version: "encryptionkey@0"
I20260926 16:41:28.165153 27217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:28.183995 27217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:28.188565 27217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:28.188865 27217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:28.208771 27217 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:34537
I20260926 16:41:28.208793 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:34537 every 8 connection(s)
I20260926 16:41:28.210397 27217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:41:28.214651 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27217
I20260926 16:41:28.215749 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:45267
--local_ip_for_outbound_sockets=127.10.110.1
--tserver_master_addrs=127.10.110.62:34537
--webserver_port=34867
--webserver_interface=127.10.110.1
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:41:28.219519 27284 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:41:28.224522 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap starting.
I20260926 16:41:28.246377 27284 log.cc:824] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:28.263298 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:41:28.263926 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Bootstrap complete.
I20260926 16:41:28.273705 27284 raft_consensus.cc:348] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:28.274715 27284 raft_consensus.cc:729] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b054271b34104531b2d1bf3f4cb75e73, State: Initialized, Role: FOLLOWER
I20260926 16:41:28.275424 27284 consensus_queue.cc:260] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:28.275676 27284 raft_consensus.cc:388] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:41:28.275796 27284 raft_consensus.cc:482] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:41:28.275996 27284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:41:28.280049 27284 raft_consensus.cc:504] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:28.280606 27284 leader_election.cc:302] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b054271b34104531b2d1bf3f4cb75e73; no voters:
I20260926 16:41:28.281286 27284 leader_election.cc:288] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:41:28.281437 27288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:41:28.282501 27288 raft_consensus.cc:686] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [term 3 LEADER]: Becoming Leader. State: Replica: b054271b34104531b2d1bf3f4cb75e73, State: Running, Role: LEADER
I20260926 16:41:28.283187 27288 consensus_queue.cc:237] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } }
I20260926 16:41:28.283531 27284 sys_catalog.cc:564] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:41:28.285755 27290 sys_catalog.cc:455] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b054271b34104531b2d1bf3f4cb75e73. Latest consensus state: current_term: 3 leader_uuid: "b054271b34104531b2d1bf3f4cb75e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } } }
I20260926 16:41:28.286156 27290 sys_catalog.cc:458] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:28.286823 27289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b054271b34104531b2d1bf3f4cb75e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b054271b34104531b2d1bf3f4cb75e73" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 34537 } } }
I20260926 16:41:28.287120 27289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:28.292271 27296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:41:28.302729 27296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:41:28.304690 27296 catalog_manager.cc:1259] Loaded cluster ID: f03eda8b4da048fb9a0ce1006cfcbd51
I20260926 16:41:28.304924 27296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:41:28.309574 27296 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:41:28.310979 27296 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Loaded TSK: 0
I20260926 16:41:28.311125 27296 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b054271b34104531b2d1bf3f4cb75e73: Loaded TSK: 1
F20260926 16:41:28.312117 27296 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3003f989ad google::LogMessage::Fail() at ??:0
@ 0x7f3003f9b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3003f984de google::LogMessage::Flush() at ??:0
@ 0x7f3003f99a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3007cfc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3007ceb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f30199c71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f30199c6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3019a522a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3019a520c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3019a9c5e9 std::function<>::operator()() at ??:0
@ 0x7f30199cacdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f30199c8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3019a50610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3019a50439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3006be4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3006be791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3006be7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3006bc2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3009e9c6db start_thread at ??:0
@ 0x7f3002e3261f clone at ??:0
W20260926 16:41:28.603619 27286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:28.604024 27286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:28.604187 27286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:28.611932 27286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:28.612088 27286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:28.612269 27286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:41:28.623169 27286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:45267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=34867
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:28.624701 27286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:28.626598 27286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:28.641531 27331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:28.641498 27334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:28.641564 27332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:28.642468 27286 server_base.cc:1034] running on GCE node
I20260926 16:41:28.643106 27286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:28.644150 27286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:28.645373 27286 hybrid_clock.cc:648] HybridClock initialized: now 1790440888645317 us; error 42 us; skew 500 ppm
I20260926 16:41:28.645710 27286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:28 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:28.648952 27286 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:41:28.650735 27286 webserver.cc:460] Webserver started at http://127.10.110.1:34867/ using document root <none> and password file <none>
I20260926 16:41:28.651467 27286 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:28.651580 27286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:28.657582 27286 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:41:28.660630 27341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:28.662246 27286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:28.662425 27286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "b3cf59495051424a8db92217b2ace2fa"
format_stamp: "Formatted at 2026-09-26 16:40:42 on dist-test-slave-whjp"
server_key: "674c6f5cb1e7ed0e1e278a8a4ce892a4"
server_key_iv: "3ab3491d5100aec63fd3594be55b2425"
server_key_version: "encryptionkey@0"
I20260926 16:41:28.663480 27286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:28.689939 27286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:28.694765 27286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:28.695065 27286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:28.696171 27286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:28.697645 27286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:28.697764 27286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:28.697876 27286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:28.697953 27286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:28.730820 27286 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:45267
I20260926 16:41:28.730890 27453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:45267 every 8 connection(s)
I20260926 16:41:28.732276 27286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:41:28.737318 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27286
I20260926 16:41:28.737599 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:28.735292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:47713)
0926 16:41:28.736083 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:28.736412 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":306,"threads_started":1}
I20260926 16:41:28.737979 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:46421
--local_ip_for_outbound_sockets=127.10.110.2
--tserver_master_addrs=127.10.110.62:34537
--webserver_port=35857
--webserver_interface=127.10.110.2
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:28.738358 27454 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:34537 (0 consecutive failures): Network error: Failed to ping master at 127.10.110.62:34537: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
I20260926 16:41:28.739264 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:28.738901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:51767)
0926 16:41:28.739068 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:28.739145 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 16:41:28.740191 27456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:28.739861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:52105)
0926 16:41:28.739983 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:28.740068 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 16:41:28.740435 27454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:41:29.120304 27457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:29.120661 27457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:29.120819 27457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:29.128680 27457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:29.128906 27457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:29.129123 27457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:41:29.140327 27457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:46421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=35857
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:29.142125 27457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:29.143814 27457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:29.158548 27462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:29.158710 27465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:29.158735 27463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:29.159492 27457 server_base.cc:1034] running on GCE node
I20260926 16:41:29.160485 27457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:29.161883 27457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:29.163194 27457 hybrid_clock.cc:648] HybridClock initialized: now 1790440889163128 us; error 71 us; skew 500 ppm
I20260926 16:41:29.163618 27457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:29 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440889, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:29.167030 27457 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:41:29.168862 27457 webserver.cc:460] Webserver started at http://127.10.110.2:35857/ using document root <none> and password file <none>
I20260926 16:41:29.169605 27457 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:29.169719 27457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:29.176941 27457 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:41:29.180486 27472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:29.182435 27457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:41:29.182654 27457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "363ce0165ab346b68cb4bb49e48bf529"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "1a85bcc21ee26664e9bdadcaa7659653"
server_key_iv: "30304473f96b3da3fade3ee0c690bfc1"
server_key_version: "encryptionkey@0"
I20260926 16:41:29.184387 27457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:29.209669 27457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:29.216105 27457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:29.216382 27457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:29.217468 27457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:29.218993 27457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:29.219101 27457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:29.219197 27457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:29.219260 27457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:29.255510 27457 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:46421
I20260926 16:41:29.255610 27584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:46421 every 8 connection(s)
I20260926 16:41:29.257087 27457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:41:29.262763 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.260600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:56541)
0926 16:41:29.261469 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.261750 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":317,"threads_started":1}
W20260926 16:41:29.263603 27585 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:34537 (0 consecutive failures): Network error: Failed to ping master at 127.10.110.62:34537: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
I20260926 16:41:29.264680 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27457
I20260926 16:41:29.264709 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.264249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:48757)
0926 16:41:29.264448 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.264546 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:41:29.265329 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:38107
--local_ip_for_outbound_sockets=127.10.110.3
--tserver_master_addrs=127.10.110.62:34537
--webserver_port=40925
--webserver_interface=127.10.110.3
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:41:29.265786 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.265349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:44895)
0926 16:41:29.265505 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.265601 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:41:29.266053 27585 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:41:29.742174 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.741125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:37203)
0926 16:41:29.741652 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.741755 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":255,"threads_started":1}
W20260926 16:41:29.747561 27588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:29.748234 27588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:29.748395 27588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:29.757205 27588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:29.757345 27588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:29.757498 27588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:41:29.769126 27588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:38107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=40925
--tserver_master_addrs=127.10.110.62:34537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:29.770521 27588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:29.772018 27588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:29.787281 27594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:29.787565 27595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:29.787689 27597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:29.788988 27588 server_base.cc:1034] running on GCE node
I20260926 16:41:29.789733 27588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:29.790913 27588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:29.792160 27588 hybrid_clock.cc:648] HybridClock initialized: now 1790440889792100 us; error 50 us; skew 500 ppm
I20260926 16:41:29.792522 27588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:29 dist-test-slave-whjp krb5kdc[26133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440889, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:29.796150 27588 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:41:29.797914 27588 webserver.cc:460] Webserver started at http://127.10.110.3:40925/ using document root <none> and password file <none>
I20260926 16:41:29.798676 27588 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:29.798815 27588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:29.805496 27588 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:41:29.808786 27604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:29.811290 27588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:41:29.811506 27588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "0d56d0f8839b4147a39b4293c8480238"
format_stamp: "Formatted at 2026-09-26 16:40:43 on dist-test-slave-whjp"
server_key: "5a38c07d7d37717e4d19f32f2fc4dd4c"
server_key_iv: "d6140bed81b4c3d60c00c86044717ed4"
server_key_version: "encryptionkey@0"
I20260926 16:41:29.812683 27588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:29.835685 27588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:29.840920 27588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:29.841238 27588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:29.842327 27588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:29.843904 27588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:29.844022 27588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:29.844143 27588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:29.844228 27588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:29.877828 27588 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:38107
I20260926 16:41:29.877876 27716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:38107 every 8 connection(s)
I20260926 16:41:29.879329 27588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:41:29.884392 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.882358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:34727)
0926 16:41:29.883156 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.883421 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":304,"threads_started":1}
W20260926 16:41:29.885317 27717 heartbeater.cc:644] Failed to heartbeat to 127.10.110.62:34537 (0 consecutive failures): Network error: Failed to ping master at 127.10.110.62:34537: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
I20260926 16:41:29.886322 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.885877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:49177)
0926 16:41:29.886076 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.886158 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 16:41:29.887399 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:29.886999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:38119)
0926 16:41:29.887189 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:29.887273 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 16:41:29.887684 27717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:41:29.889379 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27588
I20260926 16:41:30.268127 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:30.266783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:39413)
0926 16:41:30.267569 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:30.267690 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":450,"threads_started":1}
I20260926 16:41:30.744283 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:30.743365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:45699)
0926 16:41:30.743979 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:30.744097 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":302,"threads_started":1}
I20260926 16:41:30.889230 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:30.888339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:34307)
0926 16:41:30.888840 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:30.888939 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260926 16:41:31.269850 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:31.269143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:54959)
0926 16:41:31.269583 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:31.269670 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":207,"threads_started":1}
I20260926 16:41:31.746271 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:31.745449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:33213)
0926 16:41:31.745967 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:31.746082 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":286,"threads_started":1}
I20260926 16:41:31.891131 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:31.890287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:49319)
0926 16:41:31.890849 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:31.890946 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":252,"threads_started":1}
I20260926 16:41:32.271766 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:32.270911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:39951)
0926 16:41:32.271455 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:32.271581 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":300,"threads_started":1}
I20260926 16:41:32.748222 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:32.747334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:42465)
0926 16:41:32.747878 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:32.747988 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260926 16:41:32.892940 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:32.892210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:33993)
0926 16:41:32.892671 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:32.892760 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":221,"threads_started":1}
I20260926 16:41:33.273722 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:33.272937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:59137)
0926 16:41:33.273461 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:33.273550 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":301,"threads_started":1}
I20260926 16:41:33.750195 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:33.749316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:45407)
0926 16:41:33.749889 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:33.749981 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":309,"threads_started":1}
I20260926 16:41:33.894680 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:33.893866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:54895)
0926 16:41:33.894404 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:33.894491 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":303,"threads_started":1}
I20260926 16:41:34.275820 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:34.274956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:45453)
0926 16:41:34.275508 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:34.275636 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
I20260926 16:41:34.752264 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:34.751292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:46699)
0926 16:41:34.751934 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:34.752070 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":315,"threads_started":1}
I20260926 16:41:34.896559 27735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:34.895714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:53733)
0926 16:41:34.896288 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:34.896376 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":322,"threads_started":1}
I20260926 16:41:35.277931 27736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:35.277131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:48273)
0926 16:41:35.277650 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:35.277762 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260926 16:41:35.754333 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:35.753423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:51017)
0926 16:41:35.754017 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:35.754117 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":239,"threads_started":1}
I20260926 16:41:35.898722 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:35.897779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:56771)
0926 16:41:35.898424 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:35.898544 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":347,"threads_started":1}
I20260926 16:41:36.280055 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:36.279022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:40115)
0926 16:41:36.279626 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:36.279770 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":294,"threads_started":1}
I20260926 16:41:36.756434 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:36.755548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:40965)
0926 16:41:36.756114 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:36.756251 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":303,"threads_started":1}
I20260926 16:41:36.900946 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:36.899991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:51527)
0926 16:41:36.900632 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:36.900720 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":300,"threads_started":1}
I20260926 16:41:37.281735 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:37.280969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:56295)
0926 16:41:37.281484 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:37.281564 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":298,"threads_started":1}
I20260926 16:41:37.758658 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:37.757733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:38995)
0926 16:41:37.758368 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:37.758468 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":324,"threads_started":1}
I20260926 16:41:37.902908 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:37.902153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:58391)
0926 16:41:37.902636 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:37.902737 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260926 16:41:38.283713 27745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:38.282835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:44407)
0926 16:41:38.283422 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:38.283518 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":341,"threads_started":1}
I20260926 16:41:38.760735 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:38.759786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:50957)
0926 16:41:38.760425 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:38.760522 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":393,"threads_started":1}
I20260926 16:41:38.904946 27747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:38.904069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:60535)
0926 16:41:38.904668 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:38.904767 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":331,"threads_started":1}
I20260926 16:41:39.285825 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:39.284986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:38059)
0926 16:41:39.285512 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:39.285606 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
I20260926 16:41:39.762830 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:39.761858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:57025)
0926 16:41:39.762491 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:39.762602 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":403,"threads_started":1}
I20260926 16:41:39.906955 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:39.906083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:35545)
0926 16:41:39.906679 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:39.906784 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":302,"threads_started":1}
I20260926 16:41:40.288372 27751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:40.286998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:39321)
0926 16:41:40.287657 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:40.288147 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":249,"threads_started":1}
I20260926 16:41:40.764619 27752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:40.763890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:34775)
0926 16:41:40.764360 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:40.764448 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260926 16:41:40.909070 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:40.908144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:57789)
0926 16:41:40.908767 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:40.908876 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":372,"threads_started":1}
I20260926 16:41:41.290508 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:41.289693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:43173)
0926 16:41:41.290220 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:41.290333 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260926 16:41:41.766871 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:41.765863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:40323)
0926 16:41:41.766507 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:41.766644 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":341,"threads_started":1}
I20260926 16:41:41.911031 27756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:41.910158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:39241)
0926 16:41:41.910733 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:41.910830 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":288,"threads_started":1}
I20260926 16:41:42.292665 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:42.291794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:38203)
0926 16:41:42.292355 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:42.292466 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":262,"threads_started":1}
I20260926 16:41:42.768909 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:42.768087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:40259)
0926 16:41:42.768599 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:42.768715 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
I20260926 16:41:42.913004 27759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:42.912192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:48453)
0926 16:41:42.912711 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:42.912813 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":230,"threads_started":1}
I20260926 16:41:43.294575 27760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:43.293786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:49533)
0926 16:41:43.294286 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:43.294383 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260926 16:41:43.771133 27761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:43.770226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:59461)
0926 16:41:43.770742 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:43.770889 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":246,"threads_started":1}
I20260926 16:41:43.915352 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:43.914269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:54065)
0926 16:41:43.914909 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:43.915148 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":312,"threads_started":1}
I20260926 16:41:44.296821 27763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:44.295951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.2:34089)
0926 16:41:44.296470 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:44.296617 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":265,"threads_started":1}
I20260926 16:41:44.773396 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:44.772489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.1:56123)
0926 16:41:44.773099 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:44.773215 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260926 16:41:44.892313 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 27286
I20260926 16:41:44.918076 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:44.916745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:34537 (local address 127.10.110.3:45057)
0926 16:41:44.917623 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:44.917809 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.62:34537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":639,"threads_started":1}
I20260926 16:41:44.920008 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:44.894049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.10.110.1:45267)
0926 16:41:44.894851 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:44.894866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:44.894913 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:41:44.896797 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:44.896809 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:44.897289 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:44.897644 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:44.899161 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:44.900665 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:44.902026 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:44.902777 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:44.902951 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:44.910756 (+ 7805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:44.910767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:44.910788 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:44.910843 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:44.914690 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:44.915973 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:44.916001 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:44.916029 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:44.916354 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:44.916914 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:44.916919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:44.916923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:44.917891 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:44.918317 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:44.919050 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 16:41:44.919508 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":382,"threads_started":1}
I20260926 16:41:44.921592 27449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:45.016716 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 27286
I20260926 16:41:45.033154 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 27457
I20260926 16:41:45.055281 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:45.034266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.10.110.2:46421)
0926 16:41:45.034805 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:45.034821 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:45.034860 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:41:45.036844 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:45.036856 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:45.037354 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:45.037722 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:45.039276 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:45.040755 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:45.042082 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:45.042873 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:45.043074 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:45.045874 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:45.045883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:45.045901 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:45.045949 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:45.049121 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:45.049946 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:45.049954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:45.049962 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:45.051503 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:45.052371 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:45.052376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:45.052379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:45.053261 (+ 882us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:45.053853 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:45.054426 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 16:41:45.054967 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":230,"threads_started":1}
I20260926 16:41:45.056995 27580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:45.146580 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 27457
I20260926 16:41:45.164906 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 27588
I20260926 16:41:45.186474 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:45.166366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.10.110.3:38107)
0926 16:41:45.167089 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:45.167104 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:45.167150 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:41:45.169009 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:45.169023 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:45.169541 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:45.169901 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:45.171449 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:45.173057 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:45.174553 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:45.175391 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:45.175555 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:45.178934 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:45.178944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:45.178964 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:45.179017 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:45.182444 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:45.183418 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:45.183427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:45.183435 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:45.183734 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:45.184237 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:45.184241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:45.184243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:45.184949 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:45.185216 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:45.185700 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 16:41:45.186149 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":345,"threads_started":1}
I20260926 16:41:45.188058 27712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:45.280717 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 27588
2026-09-26T16:41:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64855 ms)
[----------] 30 tests from SecurityITest (289920 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:41:45 dist-test-slave-whjp krb5kdc[27774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:41:45 dist-test-slave-whjp krb5kdc[27774](info): set up 2 sockets
Sep 26 16:41:45 dist-test-slave-whjp krb5kdc[27774](info): commencing operation
krb5kdc: starting...
W20260926 16:41:46.357070 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:41:46 dist-test-slave-whjp krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:41:46.629319 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40929
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:40929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:46.993438 27787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:46.993803 27787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:46.993952 27787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:47.002408 27787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:41:47.002565 27787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:47.002647 27787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:47.002731 27787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:41:47.002789 27787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:41:47.015403 27787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:40929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:47.017184 27787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:47.018818 27787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:47.034629 27792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:47.034592 27795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:47.034678 27793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:47.037920 27787 server_base.cc:1034] running on GCE node
I20260926 16:41:47.038856 27787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:47.040683 27787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:47.041965 27787 hybrid_clock.cc:648] HybridClock initialized: now 1790440907041902 us; error 54 us; skew 500 ppm
I20260926 16:41:47.042413 27787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:47 dist-test-slave-whjp krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440907, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:47.046011 27787 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:41:47.048075 27787 webserver.cc:460] Webserver started at http://127.10.110.62:42527/ using document root <none> and password file <none>
I20260926 16:41:47.048849 27787 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:47.048983 27787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:47.049317 27787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:47.052814 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "fc9640d90a1741ee876d47e63e2b4935"
format_stamp: "Formatted at 2026-09-26 16:41:47 on dist-test-slave-whjp"
server_key: "5f6ce3fa40d726c98e7f8ee3fcb2cb68"
server_key_iv: "78ec7b6fa42f34714a0bb7d4dffd929b"
server_key_version: "encryptionkey@0"
I20260926 16:41:47.053658 27787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "fc9640d90a1741ee876d47e63e2b4935"
format_stamp: "Formatted at 2026-09-26 16:41:47 on dist-test-slave-whjp"
server_key: "5f6ce3fa40d726c98e7f8ee3fcb2cb68"
server_key_iv: "78ec7b6fa42f34714a0bb7d4dffd929b"
server_key_version: "encryptionkey@0"
I20260926 16:41:47.059897 27787 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:41:47.064410 27802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:47.066125 27787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:41:47.066339 27787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "fc9640d90a1741ee876d47e63e2b4935"
format_stamp: "Formatted at 2026-09-26 16:41:47 on dist-test-slave-whjp"
server_key: "5f6ce3fa40d726c98e7f8ee3fcb2cb68"
server_key_iv: "78ec7b6fa42f34714a0bb7d4dffd929b"
server_key_version: "encryptionkey@0"
I20260926 16:41:47.066566 27787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:47.094786 27787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:47.099750 27787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:47.100090 27787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:47.119894 27787 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:40929
I20260926 16:41:47.119884 27853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:40929 every 8 connection(s)
I20260926 16:41:47.121502 27787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:41:47.125666 27854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:47.125782 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27787
I20260926 16:41:47.125981 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:41:47.126621 10680 external_mini_cluster.cc:1428] Setting key 7546c9d06afd0ce3a455a4c9d698e142
I20260926 16:41:47.141453 27854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935: Bootstrap starting.
Sep 26 16:41:47 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:47.145903 27854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:47.147213 27854 log.cc:824] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:47.151072 27854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935: No bootstrap required, opened a new log
I20260926 16:41:47.155781 27857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:47.129078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.10.110.62:40929)
0926 16:41:47.130117 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:47.130134 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:47.130183 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:41:47.132561 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:47.132596 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:47.133153 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:47.133595 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:47.135712 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:47.137236 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:47.139302 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:47.140277 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:47.140453 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:47.145803 (+ 5350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:47.145813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:47.145830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:47.145888 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:47.149739 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:47.150940 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:47.150948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:47.150955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:47.151255 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:47.151851 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:47.151858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:47.151861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:47.152549 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:47.152920 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:47.153545 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 16:41:47.154092 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":499,"threads_started":1}
I20260926 16:41:47.160379 27854 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9640d90a1741ee876d47e63e2b4935" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40929 } }
I20260926 16:41:47.160938 27854 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:47.161072 27854 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9640d90a1741ee876d47e63e2b4935, State: Initialized, Role: FOLLOWER
I20260926 16:41:47.161731 27854 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9640d90a1741ee876d47e63e2b4935" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40929 } }
I20260926 16:41:47.162149 27854 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:41:47.162415 27854 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:41:47.162627 27854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:47.165109 27854 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9640d90a1741ee876d47e63e2b4935" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40929 } }
I20260926 16:41:47.165756 27854 leader_election.cc:302] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc9640d90a1741ee876d47e63e2b4935; no voters:
I20260926 16:41:47.166376 27854 leader_election.cc:288] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:41:47.166527 27859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:47.167194 27859 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [term 1 LEADER]: Becoming Leader. State: Replica: fc9640d90a1741ee876d47e63e2b4935, State: Running, Role: LEADER
I20260926 16:41:47.167802 27859 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9640d90a1741ee876d47e63e2b4935" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40929 } }
I20260926 16:41:47.168610 27854 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:41:47.170711 27861 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc9640d90a1741ee876d47e63e2b4935. Latest consensus state: current_term: 1 leader_uuid: "fc9640d90a1741ee876d47e63e2b4935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9640d90a1741ee876d47e63e2b4935" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40929 } } }
I20260926 16:41:47.171028 27861 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:47.171993 27865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:41:47.171756 27860 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc9640d90a1741ee876d47e63e2b4935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9640d90a1741ee876d47e63e2b4935" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40929 } } }
I20260926 16:41:47.172165 27860 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:47.177434 27865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:41:47.187284 27865 catalog_manager.cc:1347] Generated new cluster ID: 10381e21d8e24ceeb600c8041448a428
I20260926 16:41:47.187423 27865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:41:47.246774 27865 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:41:47.248626 27865 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:41:47.256325 27865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935: Generated new TSK 0
I20260926 16:41:47.257643 27865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:41:47.321393 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40929
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:47.680912 27880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:47.681244 27880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:47.681347 27880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:47.690129 27880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:47.690310 27880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:47.690436 27880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:41:47.701632 27880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:47.703317 27880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:47.705305 27880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:47.721617 27885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:47.721689 27888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:47.721872 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:47.722483 27880 server_base.cc:1034] running on GCE node
I20260926 16:41:47.723125 27880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:47.724203 27880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:47.725405 27880 hybrid_clock.cc:648] HybridClock initialized: now 1790440907725338 us; error 57 us; skew 500 ppm
I20260926 16:41:47.725719 27880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:47 dist-test-slave-whjp krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440907, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:47.729310 27880 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:41:47.731077 27880 webserver.cc:460] Webserver started at http://127.10.110.1:46343/ using document root <none> and password file <none>
I20260926 16:41:47.731817 27880 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:47.732004 27880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:47.732343 27880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:47.735808 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "595af71c1ab44b1f98968eeaa3d210f1"
format_stamp: "Formatted at 2026-09-26 16:41:47 on dist-test-slave-whjp"
server_key: "7a6b205138440ca386625a053535c141"
server_key_iv: "a0356c793d01212ead4f83a8afc1f7b0"
server_key_version: "encryptionkey@0"
I20260926 16:41:47.736770 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "595af71c1ab44b1f98968eeaa3d210f1"
format_stamp: "Formatted at 2026-09-26 16:41:47 on dist-test-slave-whjp"
server_key: "7a6b205138440ca386625a053535c141"
server_key_iv: "a0356c793d01212ead4f83a8afc1f7b0"
server_key_version: "encryptionkey@0"
I20260926 16:41:47.743386 27880 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:41:47.747634 27895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:47.749275 27880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:47.749457 27880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "595af71c1ab44b1f98968eeaa3d210f1"
format_stamp: "Formatted at 2026-09-26 16:41:47 on dist-test-slave-whjp"
server_key: "7a6b205138440ca386625a053535c141"
server_key_iv: "a0356c793d01212ead4f83a8afc1f7b0"
server_key_version: "encryptionkey@0"
I20260926 16:41:47.749688 27880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:47.770995 27880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:47.776202 27880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:47.776479 27880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:47.777613 27880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:47.779160 27880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:47.779241 27880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:47.779361 27880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:47.779417 27880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:47.813958 27880 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:41665
I20260926 16:41:47.814013 28007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:41665 every 8 connection(s)
I20260926 16:41:47.815368 27880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:41:47.817816 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 27880
I20260926 16:41:47.818006 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:41:47.818791 10680 external_mini_cluster.cc:1428] Setting key 50410a7b126e2689ac48702f1f1feb6b
Sep 26 16:41:47 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440907, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:47.840387 28010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:47.818092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:41997 (local address 127.10.110.62:40929)
0926 16:41:47.818651 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:47.818658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:47.819846 (+ 1188us) server_negotiation.cc:407] Connection header received
0926 16:41:47.821831 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:47.821837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:47.821928 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:47.822059 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:47.824091 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:47.825208 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:47.827454 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:47.828064 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:47.828224 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:47.833474 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:47.833480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:47.833484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:47.833527 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:47.836261 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:47.837424 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:47.837431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:47.837433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:47.837645 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:47.838242 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:47.838246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:47.838249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:47.838569 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:47.838712 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:47.839747 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0926 16:41:47.840098 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":251,"threads_started":1}
I20260926 16:41:47.841877 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:47.818762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40929 (local address 127.10.110.1:41997)
0926 16:41:47.819632 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:47.819688 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:47.821513 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:47.822492 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:47.822502 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:47.823174 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:47.823875 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:47.823906 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:47.825428 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:47.825433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:47.827211 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:47.827231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:47.828382 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:47.828391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:47.828751 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:47.829845 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:47.829893 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:47.833134 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:47.836493 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:47.836504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:47.836528 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:47.837231 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:47.837767 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:47.837771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:47.837775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:47.838061 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:47.838715 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:47.838725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:47.839254 (+ 529us) client_negotiation.cc:769] Sending connection context
0926 16:41:47.839593 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 16:41:47.840283 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":359,"threads_started":1}
I20260926 16:41:47.844363 28008 heartbeater.cc:347] Connected to a master server at 127.10.110.62:40929
I20260926 16:41:47.844956 28008 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:41:47.846319 28008 heartbeater.cc:511] Master 127.10.110.62:40929 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:41:47.849236 27819 ts_manager.cc:194] Registered new tserver with Master: 595af71c1ab44b1f98968eeaa3d210f1 (127.10.110.1:41665)
I20260926 16:41:47.850950 27819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:41997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:41:47.880204 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40929
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:48.244391 28014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:48.244704 28014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:48.244828 28014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:48.253021 28014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:48.253167 28014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:48.253311 28014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:41:48.263942 28014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:48.265373 28014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:48.266781 28014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:48.282373 28019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:48.282399 28020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:48.283057 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:48.283517 28014 server_base.cc:1034] running on GCE node
I20260926 16:41:48.284268 28014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:48.285385 28014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:48.286674 28014 hybrid_clock.cc:648] HybridClock initialized: now 1790440908286582 us; error 82 us; skew 500 ppm
I20260926 16:41:48.287039 28014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:48 dist-test-slave-whjp krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:48.290900 28014 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:41:48.292598 28014 webserver.cc:460] Webserver started at http://127.10.110.2:33449/ using document root <none> and password file <none>
I20260926 16:41:48.293354 28014 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:48.293488 28014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:48.293816 28014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:48.297092 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "58dd063014a94177a6e753a8336e922a"
format_stamp: "Formatted at 2026-09-26 16:41:48 on dist-test-slave-whjp"
server_key: "1ceaca592f811d9bfeb9b022cf4c470a"
server_key_iv: "dd8c87c47ff9257c2fd60b6b7c5c031d"
server_key_version: "encryptionkey@0"
I20260926 16:41:48.297971 28014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "58dd063014a94177a6e753a8336e922a"
format_stamp: "Formatted at 2026-09-26 16:41:48 on dist-test-slave-whjp"
server_key: "1ceaca592f811d9bfeb9b022cf4c470a"
server_key_iv: "dd8c87c47ff9257c2fd60b6b7c5c031d"
server_key_version: "encryptionkey@0"
I20260926 16:41:48.303752 28014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:41:48.307787 28029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:48.309585 28014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:48.309785 28014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "58dd063014a94177a6e753a8336e922a"
format_stamp: "Formatted at 2026-09-26 16:41:48 on dist-test-slave-whjp"
server_key: "1ceaca592f811d9bfeb9b022cf4c470a"
server_key_iv: "dd8c87c47ff9257c2fd60b6b7c5c031d"
server_key_version: "encryptionkey@0"
I20260926 16:41:48.309993 28014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:48.337924 28014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:48.343281 28014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:48.343608 28014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:48.344779 28014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:48.346323 28014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:48.346421 28014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:48.346551 28014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:48.346608 28014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:48.387398 28014 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:34507
I20260926 16:41:48.387449 28141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:34507 every 8 connection(s)
I20260926 16:41:48.388787 28014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:41:48.396289 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28014
I20260926 16:41:48.396484 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:41:48.396986 10680 external_mini_cluster.cc:1428] Setting key 36c0e07305ab37b1d4939a08e5666d20
Sep 26 16:41:48 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:48.414860 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:48.391605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:46725 (local address 127.10.110.62:40929)
0926 16:41:48.392208 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:48.392214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:48.393401 (+ 1187us) server_negotiation.cc:407] Connection header received
0926 16:41:48.395818 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:48.395825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:48.395965 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:48.396074 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:48.398130 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:48.399146 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:48.401961 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:48.402466 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:48.402599 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:48.407564 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:48.407571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:48.407574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:48.407627 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:48.411132 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:48.412159 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:48.412164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:48.412166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:48.412344 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:48.412854 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:48.412858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:48.412860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:48.413175 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:48.413325 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:48.414176 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 16:41:48.414444 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":297,"threads_started":1}
I20260926 16:41:48.416188 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:48.392309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40929 (local address 127.10.110.2:46725)
0926 16:41:48.393194 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:48.393246 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:48.395408 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:48.396410 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:48.396422 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:48.397097 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:48.397919 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:48.397945 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:48.399369 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:48.399374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:48.401624 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:48.401652 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:48.402774 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:48.402782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:48.403111 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:48.404142 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:48.404174 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:48.407234 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:48.411365 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:48.411373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:48.411391 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:48.411983 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:48.412485 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:48.412488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:48.412490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:48.412716 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:48.413330 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:48.413339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:48.413753 (+ 414us) client_negotiation.cc:769] Sending connection context
0926 16:41:48.414030 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 16:41:48.414652 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":345,"threads_started":1}
I20260926 16:41:48.417944 28142 heartbeater.cc:347] Connected to a master server at 127.10.110.62:40929
I20260926 16:41:48.418345 28142 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:48.419462 28142 heartbeater.cc:511] Master 127.10.110.62:40929 requested a full tablet report, sending...
I20260926 16:41:48.421761 27819 ts_manager.cc:194] Registered new tserver with Master: 58dd063014a94177a6e753a8336e922a (127.10.110.2:34507)
I20260926 16:41:48.422691 27819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:46725
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:41:48.464906 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40929
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:41:48.854699 28008 heartbeater.cc:503] Master 127.10.110.62:40929 was elected leader, sending a full tablet report...
W20260926 16:41:48.870193 28148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:48.870518 28148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:48.870664 28148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:48.878248 28148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:48.878373 28148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:48.878509 28148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:41:48.888728 28148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:40929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:48.890066 28148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:48.891510 28148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:48.906919 28153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:48.907106 28156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:48.907191 28154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:48.908459 28148 server_base.cc:1034] running on GCE node
I20260926 16:41:48.909381 28148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:48.910507 28148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:48.911765 28148 hybrid_clock.cc:648] HybridClock initialized: now 1790440908911696 us; error 66 us; skew 500 ppm
I20260926 16:41:48.912248 28148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:48 dist-test-slave-whjp krb5kdc[27774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:48.916388 28148 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:41:48.918375 28148 webserver.cc:460] Webserver started at http://127.10.110.3:41267/ using document root <none> and password file <none>
I20260926 16:41:48.919198 28148 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:48.919389 28148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:48.919785 28148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:48.923295 28148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "57994c753cc6476d9fe3c6a14c6474e8"
format_stamp: "Formatted at 2026-09-26 16:41:48 on dist-test-slave-whjp"
server_key: "72ff00e1ffcb62daeebfdb3d88bdd2df"
server_key_iv: "dce804df48397bb140bc680a1df13e5c"
server_key_version: "encryptionkey@0"
I20260926 16:41:48.924219 28148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "57994c753cc6476d9fe3c6a14c6474e8"
format_stamp: "Formatted at 2026-09-26 16:41:48 on dist-test-slave-whjp"
server_key: "72ff00e1ffcb62daeebfdb3d88bdd2df"
server_key_iv: "dce804df48397bb140bc680a1df13e5c"
server_key_version: "encryptionkey@0"
I20260926 16:41:48.930012 28148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:41:48.934221 28163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:48.935946 28148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:48.936134 28148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "57994c753cc6476d9fe3c6a14c6474e8"
format_stamp: "Formatted at 2026-09-26 16:41:48 on dist-test-slave-whjp"
server_key: "72ff00e1ffcb62daeebfdb3d88bdd2df"
server_key_iv: "dce804df48397bb140bc680a1df13e5c"
server_key_version: "encryptionkey@0"
I20260926 16:41:48.936347 28148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:48.953934 28148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:48.958710 28148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:48.958998 28148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:48.960064 28148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:48.961527 28148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:48.961609 28148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:48.961735 28148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:48.961787 28148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:48.996762 28148 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:42491
I20260926 16:41:48.996842 28275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:42491 every 8 connection(s)
I20260926 16:41:48.998498 28148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:41:49.003448 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28148
I20260926 16:41:49.003924 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:41:49.004796 10680 external_mini_cluster.cc:1428] Setting key 58d52acbd5e148f0c495f117a297f8f5
Sep 26 16:41:49 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:49.027570 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.001463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:40219 (local address 127.10.110.62:40929)
0926 16:41:49.001970 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.001975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.003518 (+ 1543us) server_negotiation.cc:407] Connection header received
0926 16:41:49.007121 (+ 3603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.007128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.007261 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.007369 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:49.009546 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.010547 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.012781 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.013481 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.013753 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.018208 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:49.018214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:49.018217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:49.018259 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:49.022405 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.023564 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.023570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.023573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.023743 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.024394 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.024399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.024401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.024865 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:49.025036 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.026885 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.027110 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
I20260926 16:41:49.028457 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.001978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:40929 (local address 127.10.110.3:40219)
0926 16:41:49.003259 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.003325 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:49.006671 (+ 3346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:49.007719 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:49.007735 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:49.008485 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:49.009294 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.009331 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.010797 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.010801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.012569 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.012588 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.013871 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.013879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.014203 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.015183 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:49.015213 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:49.017804 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:49.022706 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:49.022719 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:49.022742 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:49.023393 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:49.023938 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:49.023944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:49.023949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:49.024226 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:49.025265 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:49.025276 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:49.026016 (+ 740us) client_negotiation.cc:769] Sending connection context
0926 16:41:49.026673 (+ 657us) client_negotiation.cc:240] Negotiation successful
0926 16:41:49.027207 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":425,"threads_started":1}
I20260926 16:41:49.030779 28276 heartbeater.cc:347] Connected to a master server at 127.10.110.62:40929
I20260926 16:41:49.031288 28276 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:49.032231 28276 heartbeater.cc:511] Master 127.10.110.62:40929 requested a full tablet report, sending...
I20260926 16:41:49.034293 27819 ts_manager.cc:194] Registered new tserver with Master: 57994c753cc6476d9fe3c6a14c6474e8 (127.10.110.3:42491)
I20260926 16:41:49.035455 27819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:40219
I20260926 16:41:49.037051 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:41:49.057229 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.041877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.10.110.62:40929)
0926 16:41:49.042056 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.042061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.042593 (+ 532us) server_negotiation.cc:407] Connection header received
0926 16:41:49.042742 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.042746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.042832 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.042929 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:49.044528 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.045356 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.047128 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.047597 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.047776 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.050895 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:49.050901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:49.050905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:49.050948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:49.053993 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.054938 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.054945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.054949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.055157 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.055648 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.055653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.055656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.056085 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:49.056322 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.056790 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.056991 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:41:49.062448 27819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:41:49.071055 27819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:41:49.111433 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.086106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.10.110.1:41665)
0926 16:41:49.101430 (+ 15324us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.101446 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.101482 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:41:49.101606 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.101625 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.101936 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.102154 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:49.103419 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.105019 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.107989 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.109963 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.110155 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.110437 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.110864 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.111065 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13867,"thread_start_us":15077,"threads_started":1}
I20260926 16:41:49.111569 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.085746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55816)
0926 16:41:49.099083 (+ 13337us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.099148 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:49.099455 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:49.102508 (+ 3053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:49.102518 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:49.102564 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:49.103246 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.103262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.105189 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.105197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.107471 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.107497 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.110226 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.110233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.110517 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.110627 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 16:41:49.110790 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 16:41:49.111017 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12956,"mutex_wait_us":43,"spinlock_wait_cycles":896,"thread_start_us":14096,"threads_started":1}
I20260926 16:41:49.111699 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.086272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.10.110.2:34507)
0926 16:41:49.101882 (+ 15610us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.101889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.101918 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:41:49.102012 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.102019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.102263 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.103938 (+ 1675us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:49.104226 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.106139 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.108797 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.110011 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.110164 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.110403 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.111164 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.111358 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15448,"thread_start_us":15293,"threads_started":1}
I20260926 16:41:49.112989 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.085965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:34507 (local address 127.0.0.1:38668)
0926 16:41:49.101152 (+ 15187us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.101173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:49.101318 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:49.102508 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:49.102518 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:49.102572 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:49.103703 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.103717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.106298 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.106304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.108593 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.108608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.110556 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.110562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.110783 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.110814 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:41:49.110889 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 16:41:49.111016 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15055,"mutex_wait_us":50,"spinlock_wait_cycles":9728,"thread_start_us":12494,"threads_started":1}
I20260926 16:41:49.114182 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.101996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:42491 (local address 127.0.0.1:50018)
0926 16:41:49.103255 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.103275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:49.105246 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:49.106513 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:49.106518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:49.106549 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:49.107005 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.107018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.109382 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.109389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.111415 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.111437 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.113331 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.113343 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.113523 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.113558 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:41:49.113651 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:41:49.113794 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":922,"threads_started":1}
I20260926 16:41:49.115109 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.100185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.10.110.3:42491)
0926 16:41:49.105847 (+ 5662us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.105856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.105891 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:41:49.105984 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.105992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.106276 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.107195 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:49.107414 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.109202 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.111661 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.112881 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.113041 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.113289 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.114534 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.114754 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5468,"thread_start_us":5265,"threads_started":1}
I20260926 16:41:49.117427 27943 tablet_service.cc:1467] Processing CreateTablet for tablet be54406d7c5f42ddaa26d7e8dae28e6d (DEFAULT_TABLE table=test-table [id=b0733c7f4281481d8f88879aaf056a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:49.117520 28077 tablet_service.cc:1467] Processing CreateTablet for tablet be54406d7c5f42ddaa26d7e8dae28e6d (DEFAULT_TABLE table=test-table [id=b0733c7f4281481d8f88879aaf056a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:49.119494 27943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be54406d7c5f42ddaa26d7e8dae28e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:49.119558 28077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be54406d7c5f42ddaa26d7e8dae28e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:49.120348 28211 tablet_service.cc:1467] Processing CreateTablet for tablet be54406d7c5f42ddaa26d7e8dae28e6d (DEFAULT_TABLE table=test-table [id=b0733c7f4281481d8f88879aaf056a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:49.122144 28211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be54406d7c5f42ddaa26d7e8dae28e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:49.141764 28297 tablet_bootstrap.cc:492] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Bootstrap starting.
I20260926 16:41:49.144379 28295 tablet_bootstrap.cc:492] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Bootstrap starting.
I20260926 16:41:49.148932 28297 tablet_bootstrap.cc:654] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:49.149508 28295 tablet_bootstrap.cc:654] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:49.150497 28296 tablet_bootstrap.cc:492] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Bootstrap starting.
I20260926 16:41:49.150615 28297 log.cc:824] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:49.152487 28295 log.cc:824] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:49.154654 28296 tablet_bootstrap.cc:654] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:49.155185 28297 tablet_bootstrap.cc:492] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: No bootstrap required, opened a new log
I20260926 16:41:49.155550 28297 ts_tablet_manager.cc:1397] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260926 16:41:49.155773 28295 tablet_bootstrap.cc:492] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: No bootstrap required, opened a new log
I20260926 16:41:49.156140 28295 ts_tablet_manager.cc:1397] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260926 16:41:49.156148 28296 log.cc:824] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:49.159888 28296 tablet_bootstrap.cc:492] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: No bootstrap required, opened a new log
I20260926 16:41:49.160240 28296 ts_tablet_manager.cc:1397] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260926 16:41:49.163913 28295 raft_consensus.cc:348] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.164352 28295 raft_consensus.cc:374] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:49.164508 28295 raft_consensus.cc:729] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dd063014a94177a6e753a8336e922a, State: Initialized, Role: FOLLOWER
I20260926 16:41:49.165139 28295 consensus_queue.cc:260] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.166293 28142 heartbeater.cc:503] Master 127.10.110.62:40929 was elected leader, sending a full tablet report...
I20260926 16:41:49.166281 28297 raft_consensus.cc:348] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.166749 28297 raft_consensus.cc:374] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:49.166834 28297 raft_consensus.cc:729] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57994c753cc6476d9fe3c6a14c6474e8, State: Initialized, Role: FOLLOWER
I20260926 16:41:49.167060 28295 ts_tablet_manager.cc:1428] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260926 16:41:49.167593 28297 consensus_queue.cc:260] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.170130 28276 heartbeater.cc:503] Master 127.10.110.62:40929 was elected leader, sending a full tablet report...
I20260926 16:41:49.171123 28297 ts_tablet_manager.cc:1428] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 16:41:49.171342 28296 raft_consensus.cc:348] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.171852 28296 raft_consensus.cc:374] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:49.171964 28296 raft_consensus.cc:729] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595af71c1ab44b1f98968eeaa3d210f1, State: Initialized, Role: FOLLOWER
I20260926 16:41:49.172642 28296 consensus_queue.cc:260] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.175776 28296 ts_tablet_manager.cc:1428] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260926 16:41:49.250743 28277 tablet.cc:2369] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:49.263098 28302 raft_consensus.cc:482] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:41:49.263406 28302 raft_consensus.cc:504] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.265393 28302 leader_election.cc:288] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 595af71c1ab44b1f98968eeaa3d210f1 (127.10.110.1:41665), 58dd063014a94177a6e753a8336e922a (127.10.110.2:34507)
I20260926 16:41:49.272843 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.265870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:50435 (local address 127.10.110.1:41665)
0926 16:41:49.266110 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.266116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.266227 (+ 111us) server_negotiation.cc:407] Connection header received
0926 16:41:49.266477 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.266482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.266567 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.266689 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:49.267155 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.268176 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.270210 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.271788 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.272022 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.272107 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.272415 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.272528 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 16:41:49.272855 28279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.265769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.10.110.3:50435)
0926 16:41:49.266104 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.266127 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:49.266307 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:49.266725 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:49.266729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:49.266758 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:49.266967 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.266978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.268313 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.268319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.270008 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.270029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.272024 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.272031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.272220 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.272259 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:41:49.272375 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:41:49.272533 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":46}
I20260926 16:41:49.273041 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.265936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:59595 (local address 127.10.110.2:34507)
0926 16:41:49.266136 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.266141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.266718 (+ 577us) server_negotiation.cc:407] Connection header received
0926 16:41:49.266873 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.266883 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.266964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.267068 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:49.267676 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.268486 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.270673 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.272099 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.272243 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.272304 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.272660 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.272799 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:41:49.273095 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.265816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:34507 (local address 127.10.110.3:59595)
0926 16:41:49.266599 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.266622 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:49.266740 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:49.267145 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:49.267150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:49.267177 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:49.267522 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.267535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.268653 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.268657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.270429 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:49.270447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.272322 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.272328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:49.272502 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.272532 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:41:49.272617 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 16:41:49.272748 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":453,"threads_started":1}
I20260926 16:41:49.274180 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be54406d7c5f42ddaa26d7e8dae28e6d" candidate_uuid: "57994c753cc6476d9fe3c6a14c6474e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dd063014a94177a6e753a8336e922a" is_pre_election: true
I20260926 16:41:49.274297 27963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be54406d7c5f42ddaa26d7e8dae28e6d" candidate_uuid: "57994c753cc6476d9fe3c6a14c6474e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595af71c1ab44b1f98968eeaa3d210f1" is_pre_election: true
I20260926 16:41:49.274711 28097 raft_consensus.cc:2454] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57994c753cc6476d9fe3c6a14c6474e8 in term 0.
I20260926 16:41:49.274799 27963 raft_consensus.cc:2454] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57994c753cc6476d9fe3c6a14c6474e8 in term 0.
I20260926 16:41:49.275491 28165 leader_election.cc:302] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57994c753cc6476d9fe3c6a14c6474e8, 595af71c1ab44b1f98968eeaa3d210f1; no voters:
I20260926 16:41:49.275978 28302 raft_consensus.cc:2781] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:41:49.276314 28302 raft_consensus.cc:482] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:41:49.276469 28302 raft_consensus.cc:3037] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:49.278837 28302 raft_consensus.cc:504] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.279503 28302 leader_election.cc:288] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [CANDIDATE]: Term 1 election: Requested vote from peers 595af71c1ab44b1f98968eeaa3d210f1 (127.10.110.1:41665), 58dd063014a94177a6e753a8336e922a (127.10.110.2:34507)
I20260926 16:41:49.280175 28097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be54406d7c5f42ddaa26d7e8dae28e6d" candidate_uuid: "57994c753cc6476d9fe3c6a14c6474e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dd063014a94177a6e753a8336e922a"
I20260926 16:41:49.280180 27963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be54406d7c5f42ddaa26d7e8dae28e6d" candidate_uuid: "57994c753cc6476d9fe3c6a14c6474e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595af71c1ab44b1f98968eeaa3d210f1"
I20260926 16:41:49.280401 28097 raft_consensus.cc:3037] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:49.280411 27963 raft_consensus.cc:3037] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:49.282950 28097 raft_consensus.cc:2454] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57994c753cc6476d9fe3c6a14c6474e8 in term 1.
I20260926 16:41:49.282959 27963 raft_consensus.cc:2454] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57994c753cc6476d9fe3c6a14c6474e8 in term 1.
I20260926 16:41:49.283635 28166 leader_election.cc:302] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57994c753cc6476d9fe3c6a14c6474e8, 58dd063014a94177a6e753a8336e922a, 595af71c1ab44b1f98968eeaa3d210f1; no voters:
I20260926 16:41:49.284082 28302 raft_consensus.cc:2781] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:49.284709 28302 raft_consensus.cc:686] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 1 LEADER]: Becoming Leader. State: Replica: 57994c753cc6476d9fe3c6a14c6474e8, State: Running, Role: LEADER
I20260926 16:41:49.285336 28302 consensus_queue.cc:237] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } }
I20260926 16:41:49.291283 27818 catalog_manager.cc:5495] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57994c753cc6476d9fe3c6a14c6474e8 (127.10.110.3). New cstate: current_term: 1 leader_uuid: "57994c753cc6476d9fe3c6a14c6474e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57994c753cc6476d9fe3c6a14c6474e8" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42491 } health_report { overall_health: HEALTHY } } }
I20260926 16:41:49.312940 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.307151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.10.110.3:42491)
0926 16:41:49.307348 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.307352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.307372 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:41:49.307525 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.307529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.307615 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.307767 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:41:49.308191 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.308969 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.310482 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.310921 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.311049 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.311722 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:41:49.312339 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:41:49.312454 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.312631 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.312721 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 16:41:49.318207 28009 tablet.cc:2369] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:49.322800 27963 raft_consensus.cc:1261] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 1 FOLLOWER]: Refusing update from remote peer 57994c753cc6476d9fe3c6a14c6474e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:49.322800 28097 raft_consensus.cc:1261] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 1 FOLLOWER]: Refusing update from remote peer 57994c753cc6476d9fe3c6a14c6474e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:49.324280 28302 consensus_queue.cc:1035] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dd063014a94177a6e753a8336e922a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:49.324625 28305 consensus_queue.cc:1035] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "595af71c1ab44b1f98968eeaa3d210f1" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:49.356318 27819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:49.356617 27819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:49.360795 27819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc9640d90a1741ee876d47e63e2b4935: Sending DeleteTablet for 3 replicas of tablet be54406d7c5f42ddaa26d7e8dae28e6d
I20260926 16:41:49.362228 28077 tablet_service.cc:1514] Processing DeleteTablet for tablet be54406d7c5f42ddaa26d7e8dae28e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:49 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:38668
I20260926 16:41:49.362233 28211 tablet_service.cc:1514] Processing DeleteTablet for tablet be54406d7c5f42ddaa26d7e8dae28e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:49 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:50018
I20260926 16:41:49.362329 27943 tablet_service.cc:1514] Processing DeleteTablet for tablet be54406d7c5f42ddaa26d7e8dae28e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:49 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:55816
I20260926 16:41:49.364926 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 27880
I20260926 16:41:49.373106 28320 tablet_replica.cc:330] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: stopping tablet replica
I20260926 16:41:49.373684 28320 raft_consensus.cc:2229] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:49.374292 28320 raft_consensus.cc:2258] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:49.375897 28318 tablet_replica.cc:330] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: stopping tablet replica
I20260926 16:41:49.376227 28319 tablet_replica.cc:330] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: stopping tablet replica
I20260926 16:41:49.376399 28318 raft_consensus.cc:2229] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:41:49.376848 28319 raft_consensus.cc:2229] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:49.377218 28318 raft_consensus.cc:2258] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:49.377501 28319 raft_consensus.cc:2258] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:49.379532 28318 ts_tablet_manager.cc:1905] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:49.379525 28320 ts_tablet_manager.cc:1905] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:41:49 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:41:49.387665 28318 ts_tablet_manager.cc:1918] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:49.388018 28318 log.cc:1196] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/be54406d7c5f42ddaa26d7e8dae28e6d
I20260926 16:41:49.388481 28319 ts_tablet_manager.cc:1905] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:49.388747 28318 ts_tablet_manager.cc:1939] T be54406d7c5f42ddaa26d7e8dae28e6d P 57994c753cc6476d9fe3c6a14c6474e8: Deleting consensus metadata
I20260926 16:41:49.390926 27805 catalog_manager.cc:4841] TS 57994c753cc6476d9fe3c6a14c6474e8 (127.10.110.3:42491): tablet be54406d7c5f42ddaa26d7e8dae28e6d (table test-table [id=b0733c7f4281481d8f88879aaf056a88]) successfully deleted
I20260926 16:41:49.396462 28319 ts_tablet_manager.cc:1918] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:49.396664 28319 log.cc:1196] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/be54406d7c5f42ddaa26d7e8dae28e6d
I20260926 16:41:49.397234 28319 ts_tablet_manager.cc:1939] T be54406d7c5f42ddaa26d7e8dae28e6d P 58dd063014a94177a6e753a8336e922a: Deleting consensus metadata
I20260926 16:41:49.399296 27805 catalog_manager.cc:4841] TS 58dd063014a94177a6e753a8336e922a (127.10.110.2:34507): tablet be54406d7c5f42ddaa26d7e8dae28e6d (table test-table [id=b0733c7f4281481d8f88879aaf056a88]) successfully deleted
I20260926 16:41:49.401860 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.366814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.10.110.1:41665)
0926 16:41:49.369477 (+ 2663us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.369485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.375163 (+ 5678us) server_negotiation.cc:407] Connection header received
0926 16:41:49.375353 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.375359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.375449 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.375544 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:49.377582 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.378970 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.381140 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.381864 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.382023 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.393334 (+ 11311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:49.393359 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:49.393379 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:49.393438 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:49.397575 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.398759 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.398770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.398774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.398953 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.399507 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.399514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.399517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.400040 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:49.400356 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.401021 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.401295 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2474}
I20260926 16:41:49.402637 28003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:49.515622 28320 ts_tablet_manager.cc:1918] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:49.515921 28320 log.cc:1196] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/be54406d7c5f42ddaa26d7e8dae28e6d
I20260926 16:41:49.516436 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 27880
I20260926 16:41:49.516506 28320 ts_tablet_manager.cc:1939] T be54406d7c5f42ddaa26d7e8dae28e6d P 595af71c1ab44b1f98968eeaa3d210f1: Deleting consensus metadata
I20260926 16:41:49.537333 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28014
W20260926 16:41:49.537333 27804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.110.1:41665 (error 108)
I20260926 16:41:49.539618 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.539033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55838)
0926 16:41:49.539224 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.539355 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:41:49.539985 27804 catalog_manager.cc:4584] TS 595af71c1ab44b1f98968eeaa3d210f1 (127.10.110.1:41665): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be54406d7c5f42ddaa26d7e8dae28e6d: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Sep 26 16:41:49 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:41:49.555761 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.538712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.10.110.2:34507)
0926 16:41:49.538916 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.538921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.538938 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:41:49.538994 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.538998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.539145 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.539270 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:49.540977 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.541842 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.543281 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.543722 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.543881 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.548687 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:49.548706 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:49.548730 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:49.548783 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:49.552439 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.553426 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.553436 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.553438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.553613 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.554035 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.554039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.554042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.554500 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:49.554808 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.555152 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.555395 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:41:49.556471 28137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:49.594179 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.593540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55852)
0926 16:41:49.593831 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.593970 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:41:49.595525 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.595057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55868)
0926 16:41:49.595229 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.595331 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:41:49.652195 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28014
I20260926 16:41:49.656363 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.655741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55880)
0926 16:41:49.656027 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.656159 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 16:41:49.657699 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.657253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55882)
0926 16:41:49.657424 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.657521 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 16:41:49.671039 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28148
Sep 26 16:41:49 dist-test-slave-whjp krb5kdc[27774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:41:49.690044 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.671969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.10.110.3:42491)
0926 16:41:49.672154 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:49.672161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:49.672296 (+ 135us) server_negotiation.cc:407] Connection header received
0926 16:41:49.672481 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:49.672487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:49.672569 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:49.672657 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:49.673993 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.674802 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.676465 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:49.676994 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:49.677139 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:49.682659 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:49.682678 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:49.682695 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:49.682747 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:49.686621 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.687530 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.687543 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.687545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.687785 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:49.688302 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:49.688307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:49.688309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:49.688724 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:49.689119 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:49.689414 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 16:41:49.689671 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:41:49.690946 28271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:49.738626 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.737992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55894)
0926 16:41:49.738278 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.738402 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:41:49.739916 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.739389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55904)
0926 16:41:49.739630 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.739719 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 16:41:49.824990 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28148
I20260926 16:41:49.845253 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 27787
I20260926 16:41:49.846709 27849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:49.966224 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:49.965680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41665 (local address 127.0.0.1:55908)
0926 16:41:49.965865 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:49.965967 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:41665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 16:41:49.966212 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 27787
2026-09-26T16:41:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4691 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T16:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:41:50Z Disabled control of system clock
I20260926 16:41:50.048600 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44417
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:44417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:41:50.420781 28330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:50.421115 28330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:50.421200 28330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:50.429153 28330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:41:50.429301 28330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:50.429340 28330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:50.429399 28330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:41:50.429443 28330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:41:50.439879 28330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:44417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:50.441277 28330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:50.443198 28330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:50.455754 28335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:50.455724 28338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:50.455683 28336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:50.456842 28330 server_base.cc:1034] running on GCE node
I20260926 16:41:50.457721 28330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:50.459332 28330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:50.460596 28330 hybrid_clock.cc:648] HybridClock initialized: now 1790440910460498 us; error 87 us; skew 500 ppm
I20260926 16:41:50.461026 28330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:50.469718 28330 webserver.cc:460] Webserver started at http://127.10.110.62:33995/ using document root <none> and password file <none>
I20260926 16:41:50.470619 28330 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:50.470752 28330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:50.471096 28330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:50.474359 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "08863e1b78db4e818b2bf72cafb2b380"
format_stamp: "Formatted at 2026-09-26 16:41:50 on dist-test-slave-whjp"
server_key: "e3b421bd0097846ff78db14cde07dea7"
server_key_iv: "d0a01425a42d5bddaecfe28bf8381a3b"
server_key_version: "encryptionkey@0"
I20260926 16:41:50.475257 28330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "08863e1b78db4e818b2bf72cafb2b380"
format_stamp: "Formatted at 2026-09-26 16:41:50 on dist-test-slave-whjp"
server_key: "e3b421bd0097846ff78db14cde07dea7"
server_key_iv: "d0a01425a42d5bddaecfe28bf8381a3b"
server_key_version: "encryptionkey@0"
I20260926 16:41:50.481170 28330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 16:41:50.485261 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:50.486931 28330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:41:50.487150 28330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "08863e1b78db4e818b2bf72cafb2b380"
format_stamp: "Formatted at 2026-09-26 16:41:50 on dist-test-slave-whjp"
server_key: "e3b421bd0097846ff78db14cde07dea7"
server_key_iv: "d0a01425a42d5bddaecfe28bf8381a3b"
server_key_version: "encryptionkey@0"
I20260926 16:41:50.487387 28330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:50.507350 28330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:50.508351 28330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:50.508651 28330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:50.528075 28330 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:44417
I20260926 16:41:50.528092 28395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:44417 every 8 connection(s)
I20260926 16:41:50.529520 28330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:41:50.533720 28396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:50.537955 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28330
I20260926 16:41:50.538146 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:41:50.538690 10680 external_mini_cluster.cc:1428] Setting key c99e0b972abdae45dda79b66f42df48d
I20260926 16:41:50.548161 28396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380: Bootstrap starting.
I20260926 16:41:50.552692 28396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:50.554000 28396 log.cc:824] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:50.554824 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:50.540964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.10.110.62:44417)
0926 16:41:50.542253 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:50.542266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:50.542314 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:41:50.544248 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:50.544279 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:50.544886 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:50.545366 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:50.546047 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:50.547404 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:50.548976 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:50.549892 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:50.550292 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:50.551905 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:50.551917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:50.551936 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:50.552767 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:50.552881 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:50.552915 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:50.553044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:50.553312 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 16:41:50.553651 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":537,"threads_started":1}
I20260926 16:41:50.557615 28396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380: No bootstrap required, opened a new log
I20260926 16:41:50.566356 28396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08863e1b78db4e818b2bf72cafb2b380" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44417 } }
I20260926 16:41:50.566859 28396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:50.566944 28396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08863e1b78db4e818b2bf72cafb2b380, State: Initialized, Role: FOLLOWER
I20260926 16:41:50.567550 28396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08863e1b78db4e818b2bf72cafb2b380" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44417 } }
I20260926 16:41:50.567806 28396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:41:50.567955 28396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:41:50.568110 28396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:50.570745 28396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08863e1b78db4e818b2bf72cafb2b380" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44417 } }
I20260926 16:41:50.571303 28396 leader_election.cc:302] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08863e1b78db4e818b2bf72cafb2b380; no voters:
I20260926 16:41:50.571970 28396 leader_election.cc:288] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:41:50.572118 28401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:50.572834 28401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [term 1 LEADER]: Becoming Leader. State: Replica: 08863e1b78db4e818b2bf72cafb2b380, State: Running, Role: LEADER
I20260926 16:41:50.573443 28401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08863e1b78db4e818b2bf72cafb2b380" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44417 } }
I20260926 16:41:50.574533 28396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:41:50.577477 28402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08863e1b78db4e818b2bf72cafb2b380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08863e1b78db4e818b2bf72cafb2b380" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44417 } } }
I20260926 16:41:50.578608 28402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:50.578280 28403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08863e1b78db4e818b2bf72cafb2b380. Latest consensus state: current_term: 1 leader_uuid: "08863e1b78db4e818b2bf72cafb2b380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08863e1b78db4e818b2bf72cafb2b380" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44417 } } }
I20260926 16:41:50.584437 28403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:50.584872 28411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:41:50.591346 28411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:41:50.601217 28411 catalog_manager.cc:1347] Generated new cluster ID: cc15bac054134e4e9beeabc5938e8f1f
I20260926 16:41:50.601363 28411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:41:50.624691 28411 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:41:50.626185 28411 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:41:50.635941 28411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380: Generated new TSK 0
I20260926 16:41:50.636899 28411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:41:50.643690 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44417
--builtin_ntp_servers=127.10.110.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:41:51.002115 28420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:51.002414 28420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:51.002525 28420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:51.010151 28420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:51.010296 28420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:51.010437 28420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:41:51.020308 28420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:51.021669 28420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:51.023515 28420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:51.037004 28425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:51.037034 28426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:51.037065 28428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:51.037856 28420 server_base.cc:1034] running on GCE node
I20260926 16:41:51.038496 28420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:51.039516 28420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:51.040745 28420 hybrid_clock.cc:648] HybridClock initialized: now 1790440911040668 us; error 67 us; skew 500 ppm
I20260926 16:41:51.041072 28420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:51.043563 28420 webserver.cc:460] Webserver started at http://127.10.110.1:44011/ using document root <none> and password file <none>
I20260926 16:41:51.044324 28420 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:51.044438 28420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:51.044757 28420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:51.047820 28420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "0f75401cf9534a24a1783d9a91868997"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "772182036f6b594e6f684214d3527004"
server_key_iv: "95f46e7991b8b6f4cf99df3b973e3c8b"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.048719 28420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "0f75401cf9534a24a1783d9a91868997"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "772182036f6b594e6f684214d3527004"
server_key_iv: "95f46e7991b8b6f4cf99df3b973e3c8b"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.054262 28420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:41:51.057996 28434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.059584 28420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 16:41:51.059868 28420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "0f75401cf9534a24a1783d9a91868997"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "772182036f6b594e6f684214d3527004"
server_key_iv: "95f46e7991b8b6f4cf99df3b973e3c8b"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.060104 28420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:51.095448 28420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:51.096330 28420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:51.096621 28420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:51.097615 28420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:51.099107 28420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:51.099195 28420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.099293 28420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:51.099365 28420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.129814 28420 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:38523
I20260926 16:41:51.129864 28546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:38523 every 8 connection(s)
I20260926 16:41:51.131210 28420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:41:51.134337 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28420
I20260926 16:41:51.134522 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:41:51.134958 10680 external_mini_cluster.cc:1428] Setting key 5d0ba829454173644542683ef9785a2e
I20260926 16:41:51.137871 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44417
--builtin_ntp_servers=127.10.110.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:41:51.144819 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:51.133920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:35691 (local address 127.10.110.62:44417)
0926 16:41:51.134511 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:51.134528 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:51.136214 (+ 1686us) server_negotiation.cc:407] Connection header received
0926 16:41:51.138156 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:51.138161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:51.138237 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:51.138329 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:51.139515 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.140296 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.141956 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.142371 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.142486 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:51.143947 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:51.143952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:51.143955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:51.143976 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:51.144017 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:51.144029 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:51.144106 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:51.144333 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:41:51.144443 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":286,"threads_started":1}
I20260926 16:41:51.146198 28550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:51.134558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44417 (local address 127.10.110.1:35691)
0926 16:41:51.136026 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:51.136078 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:51.137751 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:51.138613 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:51.138624 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:51.138733 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:51.139337 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:51.139355 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.140456 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.140460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:51.141776 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:51.141791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.142614 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.142622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:51.142864 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:51.143549 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:41:51.143551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:51.143603 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:41:51.143713 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:51.144119 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:51.144132 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:51.144139 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:41:51.144309 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 16:41:51.144777 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":317,"threads_started":1}
I20260926 16:41:51.147958 28547 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44417
I20260926 16:41:51.148366 28547 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:51.149322 28547 heartbeater.cc:511] Master 127.10.110.62:44417 requested a full tablet report, sending...
I20260926 16:41:51.151779 28361 ts_manager.cc:194] Registered new tserver with Master: 0f75401cf9534a24a1783d9a91868997 (127.10.110.1:38523)
I20260926 16:41:51.153827 28361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.1:35691
W20260926 16:41:51.468894 28551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:51.469201 28551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:51.469290 28551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:51.476768 28551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:51.476878 28551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:51.476986 28551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:41:51.486394 28551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:51.487624 28551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:51.489430 28551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:51.503337 28556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:51.503278 28557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:51.503314 28559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:51.504151 28551 server_base.cc:1034] running on GCE node
I20260926 16:41:51.504762 28551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:51.505728 28551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:51.506923 28551 hybrid_clock.cc:648] HybridClock initialized: now 1790440911506867 us; error 53 us; skew 500 ppm
I20260926 16:41:51.507246 28551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:51.509601 28551 webserver.cc:460] Webserver started at http://127.10.110.2:40163/ using document root <none> and password file <none>
I20260926 16:41:51.510332 28551 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:51.510422 28551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:51.510730 28551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:51.513796 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "9855709147eb45a3bbc578e1424268c9"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "0b448bcdedfce9981967563cb1fe970a"
server_key_iv: "edc9379ddf0a2da01ee25e76bb80fdf5"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.514586 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "9855709147eb45a3bbc578e1424268c9"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "0b448bcdedfce9981967563cb1fe970a"
server_key_iv: "edc9379ddf0a2da01ee25e76bb80fdf5"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.519942 28551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:41:51.523741 28565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.525292 28551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:51.525461 28551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "9855709147eb45a3bbc578e1424268c9"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "0b448bcdedfce9981967563cb1fe970a"
server_key_iv: "edc9379ddf0a2da01ee25e76bb80fdf5"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.525688 28551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:51.553862 28551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:51.554783 28551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:51.555217 28551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:51.556468 28551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:51.558053 28551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:51.558146 28551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.558254 28551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:51.558319 28551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.589406 28551 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:33585
I20260926 16:41:51.589486 28677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:33585 every 8 connection(s)
I20260926 16:41:51.590746 28551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:41:51.597121 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28551
I20260926 16:41:51.597299 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:41:51.597832 10680 external_mini_cluster.cc:1428] Setting key 216ea1e7c7d6c3b2334d7c169bd4bd20
I20260926 16:41:51.600680 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44417
--builtin_ntp_servers=127.10.110.20:36023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:41:51.603950 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:51.593482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:48449 (local address 127.10.110.62:44417)
0926 16:41:51.593720 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:51.593724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:51.595300 (+ 1576us) server_negotiation.cc:407] Connection header received
0926 16:41:51.597202 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:51.597209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:51.597296 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:51.597414 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:51.598708 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.599523 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.601227 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.601814 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.601938 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:51.603196 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:51.603202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:51.603206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:51.603227 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:51.603263 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:51.603276 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:51.603355 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:51.603533 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 16:41:51.603631 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260926 16:41:51.604918 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:51.594134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44417 (local address 127.10.110.2:48449)
0926 16:41:51.595042 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:51.595178 (+ 136us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:51.596913 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:51.597689 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:51.597698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:51.597791 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:51.598491 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:51.598517 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.599699 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.599704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:51.601083 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:51.601096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:51.602000 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:51.602005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:51.602224 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:51.602881 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:41:51.602884 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:51.602929 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:41:51.603011 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:51.603358 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:51.603364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:51.603370 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:41:51.603516 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 16:41:51.603860 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260926 16:41:51.606388 28678 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44417
I20260926 16:41:51.606660 28678 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:51.607388 28678 heartbeater.cc:511] Master 127.10.110.62:44417 requested a full tablet report, sending...
I20260926 16:41:51.609048 28361 ts_manager.cc:194] Registered new tserver with Master: 9855709147eb45a3bbc578e1424268c9 (127.10.110.2:33585)
I20260926 16:41:51.610112 28361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.2:48449
W20260926 16:41:51.927982 28681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:51.928260 28681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:51.928345 28681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:51.935613 28681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:51.935716 28681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:51.935822 28681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:41:51.944912 28681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:36023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:51.946148 28681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:51.948041 28681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:51.960772 28686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:51.960834 28687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:51.960786 28689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:51.961881 28681 server_base.cc:1034] running on GCE node
I20260926 16:41:51.962498 28681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:51.963546 28681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:51.964876 28681 hybrid_clock.cc:648] HybridClock initialized: now 1790440911964787 us; error 78 us; skew 500 ppm
I20260926 16:41:51.965232 28681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:51.967593 28681 webserver.cc:460] Webserver started at http://127.10.110.3:40917/ using document root <none> and password file <none>
I20260926 16:41:51.968341 28681 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:51.968447 28681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:51.968752 28681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:51.971855 28681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "b4eeac7eacf94dffbbcf52a12427e313"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "c63abf2808a362deddb02c1334569d08"
server_key_iv: "b2c76c56b952d975e4ae05a2ea7af8ae"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.972754 28681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "b4eeac7eacf94dffbbcf52a12427e313"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "c63abf2808a362deddb02c1334569d08"
server_key_iv: "b2c76c56b952d975e4ae05a2ea7af8ae"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.979207 28681 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:41:51.983048 28695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:51.984594 28681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:51.984810 28681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "b4eeac7eacf94dffbbcf52a12427e313"
format_stamp: "Formatted at 2026-09-26 16:41:51 on dist-test-slave-whjp"
server_key: "c63abf2808a362deddb02c1334569d08"
server_key_iv: "b2c76c56b952d975e4ae05a2ea7af8ae"
server_key_version: "encryptionkey@0"
I20260926 16:41:51.985021 28681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:52.002907 28681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:52.003875 28681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:52.004205 28681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:52.005275 28681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:52.006949 28681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:52.007058 28681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:52.007176 28681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:52.007251 28681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:52.039423 28681 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:33735
I20260926 16:41:52.039470 28807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:33735 every 8 connection(s)
I20260926 16:41:52.040861 28681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:41:52.046625 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28681
I20260926 16:41:52.046809 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:41:52.047433 10680 external_mini_cluster.cc:1428] Setting key ec109502228948f4f79a06391e7cb722
I20260926 16:41:52.057036 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.043535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:49505 (local address 127.10.110.62:44417)
0926 16:41:52.043782 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.043789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.045564 (+ 1775us) server_negotiation.cc:407] Connection header received
0926 16:41:52.048298 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.048308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.048410 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.048529 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:52.050200 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.051025 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.053774 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.054395 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.054473 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.056164 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:52.056171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:52.056176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:52.056213 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:52.056254 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:52.056270 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:52.056347 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.056597 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.056702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 16:41:52.058307 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.044172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44417 (local address 127.10.110.3:49505)
0926 16:41:52.045364 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:52.045428 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:52.047876 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:52.048931 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:52.048951 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:52.049077 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:52.049966 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.050016 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.051395 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.051401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.053595 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.053611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.054726 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.054731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.054935 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.055726 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:41:52.055730 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:52.055793 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:41:52.055927 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:52.056375 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:52.056386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:52.056395 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:41:52.056552 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 16:41:52.057179 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":321,"threads_started":1}
I20260926 16:41:52.059819 28808 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44417
I20260926 16:41:52.060230 28808 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:52.061305 28808 heartbeater.cc:511] Master 127.10.110.62:44417 requested a full tablet report, sending...
I20260926 16:41:52.063149 28361 ts_manager.cc:194] Registered new tserver with Master: b4eeac7eacf94dffbbcf52a12427e313 (127.10.110.3:33735)
I20260926 16:41:52.064256 28361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.3:49505
I20260926 16:41:52.066381 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:41:52.078964 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.071150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.10.110.62:44417)
0926 16:41:52.071484 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.071491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.071735 (+ 244us) server_negotiation.cc:407] Connection header received
0926 16:41:52.071971 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.071977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.072094 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.072204 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:52.072960 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.074061 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.076234 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.076911 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.077058 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.078297 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:52.078302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:52.078306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:52.078327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:52.078360 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:52.078374 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:52.078449 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.078581 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.078660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 16:41:52.084184 28361 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:41:52.094319 28361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:41:52.131065 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.106717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:33735 (local address 127.0.0.1:47968)
0926 16:41:52.119441 (+ 12724us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:52.119485 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:52.119732 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:52.121523 (+ 1791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:52.121531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:52.121589 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:52.122260 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.122279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.124527 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.124533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.127392 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.127413 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.129817 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.129824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.130164 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.130235 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 16:41:52.130430 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 16:41:52.130648 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12450,"mutex_wait_us":243,"spinlock_wait_cycles":8960,"thread_start_us":13206,"threads_started":1}
I20260926 16:41:52.132831 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.106983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.10.110.3:33735)
0926 16:41:52.120448 (+ 13465us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.120457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.120488 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:41:52.120592 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.120602 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.120902 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.121196 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:52.122524 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.124328 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.127787 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.129574 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.131630 (+ 2056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.132137 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.132236 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.132453 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13284,"thread_start_us":13112,"threads_started":1}
I20260926 16:41:52.137967 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.107169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33585 (local address 127.0.0.1:53826)
0926 16:41:52.120200 (+ 13031us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:52.120227 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:52.120467 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:52.121522 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:52.121531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:52.121591 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:52.122282 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.122292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.124527 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.124533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.127621 (+ 3088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.127635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.137085 (+ 9450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.137092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.137290 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.137322 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:41:52.137411 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 16:41:52.137561 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12883,"mutex_wait_us":268,"spinlock_wait_cycles":10624,"thread_start_us":11831,"threads_started":1}
I20260926 16:41:52.138638 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.107432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.10.110.2:33585)
0926 16:41:52.118565 (+ 11133us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.118577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.120342 (+ 1765us) server_negotiation.cc:407] Connection header received
0926 16:41:52.120663 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.120670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.120895 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.121192 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:52.122507 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.124328 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.127881 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.136889 (+ 9008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.137042 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.137300 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.137897 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.138325 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10884,"thread_start_us":11161,"threads_started":1}
I20260926 16:41:52.138139 28743 tablet_service.cc:1467] Processing CreateTablet for tablet 0f975b34f31948bdae04d124dd5e27c5 (DEFAULT_TABLE table=test-table [id=e3beef2da9bb4bdea9f93f9f6862e26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:52.140301 28743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f975b34f31948bdae04d124dd5e27c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:52.142671 28613 tablet_service.cc:1467] Processing CreateTablet for tablet 0f975b34f31948bdae04d124dd5e27c5 (DEFAULT_TABLE table=test-table [id=e3beef2da9bb4bdea9f93f9f6862e26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:52.143296 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.120631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:38523 (local address 127.0.0.1:40454)
0926 16:41:52.131328 (+ 10697us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:52.131347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:52.131462 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:52.136706 (+ 5244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:52.136711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:52.136748 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:52.136925 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.136937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.138792 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.138797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.140599 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.140621 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.142434 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.142450 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.142668 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.142705 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:41:52.142807 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:41:52.142967 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10587,"thread_start_us":12408,"threads_started":1}
I20260926 16:41:52.144191 28613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f975b34f31948bdae04d124dd5e27c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:52.144549 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.120698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.10.110.1:38523)
0926 16:41:52.135824 (+ 15126us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.136036 (+ 212us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.136071 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:41:52.136178 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.136186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.136468 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.136696 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:52.137126 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.138631 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.140802 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.142217 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.142363 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.142629 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.143859 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.144044 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14962,"thread_start_us":15147,"threads_started":1}
I20260926 16:41:52.148478 28482 tablet_service.cc:1467] Processing CreateTablet for tablet 0f975b34f31948bdae04d124dd5e27c5 (DEFAULT_TABLE table=test-table [id=e3beef2da9bb4bdea9f93f9f6862e26d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:52.150023 28482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f975b34f31948bdae04d124dd5e27c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:52.154920 28826 tablet_bootstrap.cc:492] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Bootstrap starting.
I20260926 16:41:52.156715 28827 tablet_bootstrap.cc:492] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Bootstrap starting.
I20260926 16:41:52.158267 28547 heartbeater.cc:503] Master 127.10.110.62:44417 was elected leader, sending a full tablet report...
I20260926 16:41:52.158805 28826 tablet_bootstrap.cc:654] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:52.160801 28826 log.cc:824] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:52.160969 28827 tablet_bootstrap.cc:654] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:52.162191 28827 log.cc:824] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:52.166215 28830 tablet_bootstrap.cc:492] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Bootstrap starting.
I20260926 16:41:52.167464 28827 tablet_bootstrap.cc:492] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: No bootstrap required, opened a new log
I20260926 16:41:52.167579 28826 tablet_bootstrap.cc:492] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: No bootstrap required, opened a new log
I20260926 16:41:52.167820 28827 ts_tablet_manager.cc:1397] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 16:41:52.167932 28826 ts_tablet_manager.cc:1397] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260926 16:41:52.170519 28830 tablet_bootstrap.cc:654] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:52.171891 28830 log.cc:824] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:52.175514 28830 tablet_bootstrap.cc:492] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: No bootstrap required, opened a new log
I20260926 16:41:52.175877 28830 ts_tablet_manager.cc:1397] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 16:41:52.175887 28827 raft_consensus.cc:348] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.176390 28827 raft_consensus.cc:374] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:52.176484 28827 raft_consensus.cc:729] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9855709147eb45a3bbc578e1424268c9, State: Initialized, Role: FOLLOWER
I20260926 16:41:52.177263 28827 consensus_queue.cc:260] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.177829 28826 raft_consensus.cc:348] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.178285 28826 raft_consensus.cc:374] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:52.178378 28826 raft_consensus.cc:729] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4eeac7eacf94dffbbcf52a12427e313, State: Initialized, Role: FOLLOWER
I20260926 16:41:52.178982 28826 consensus_queue.cc:260] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.180223 28808 heartbeater.cc:503] Master 127.10.110.62:44417 was elected leader, sending a full tablet report...
I20260926 16:41:52.180922 28826 ts_tablet_manager.cc:1428] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260926 16:41:52.181846 28678 heartbeater.cc:503] Master 127.10.110.62:44417 was elected leader, sending a full tablet report...
I20260926 16:41:52.182495 28827 ts_tablet_manager.cc:1428] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260926 16:41:52.187968 28830 raft_consensus.cc:348] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.188378 28830 raft_consensus.cc:374] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:52.188481 28830 raft_consensus.cc:729] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f75401cf9534a24a1783d9a91868997, State: Initialized, Role: FOLLOWER
I20260926 16:41:52.189116 28830 consensus_queue.cc:260] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.190696 28830 ts_tablet_manager.cc:1428] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
W20260926 16:41:52.292508 28809 tablet.cc:2369] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:41:52.342950 28679 tablet.cc:2369] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:41:52.383733 28548 tablet.cc:2369] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:52.412930 28834 raft_consensus.cc:482] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:41:52.413259 28834 raft_consensus.cc:504] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.414988 28834 leader_election.cc:288] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4eeac7eacf94dffbbcf52a12427e313 (127.10.110.3:33735), 9855709147eb45a3bbc578e1424268c9 (127.10.110.2:33585)
I20260926 16:41:52.422132 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.415514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:41185 (local address 127.10.110.3:33735)
0926 16:41:52.415688 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.415693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.416407 (+ 714us) server_negotiation.cc:407] Connection header received
0926 16:41:52.416620 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.416625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.416688 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.416766 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:52.417363 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.418111 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.420496 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.421284 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.421451 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.421524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.421769 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.421894 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:41:52.422235 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.415503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:33735 (local address 127.10.110.1:41185)
0926 16:41:52.416274 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:52.416290 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:52.416423 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:52.416809 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:52.416814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:52.416840 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:52.417223 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.417236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.418254 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.418258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.420307 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.420319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.421479 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.421485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.421623 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.421647 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:41:52.421728 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:41:52.421858 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"spinlock_wait_cycles":34048,"thread_start_us":442,"threads_started":1}
I20260926 16:41:52.423167 28835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.415503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33585 (local address 127.10.110.1:47671)
0926 16:41:52.416894 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:52.416911 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:52.417027 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:52.418258 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:52.418262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:52.418284 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:52.418615 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.418622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.420040 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.420044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.421350 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:52.421362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.422529 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.422536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:52.422710 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.422742 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:41:52.422826 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:41:52.422936 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":27,"thread_start_us":1038,"threads_started":1}
I20260926 16:41:52.423281 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.415446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:47671 (local address 127.10.110.2:33585)
0926 16:41:52.415673 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.415680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.417004 (+ 1324us) server_negotiation.cc:407] Connection header received
0926 16:41:52.417144 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.417150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.417239 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.417330 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:52.418784 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.419889 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.421543 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.422341 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.422495 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.422560 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.422870 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.423001 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:41:52.423420 28763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f975b34f31948bdae04d124dd5e27c5" candidate_uuid: "0f75401cf9534a24a1783d9a91868997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4eeac7eacf94dffbbcf52a12427e313" is_pre_election: true
I20260926 16:41:52.424014 28763 raft_consensus.cc:2454] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f75401cf9534a24a1783d9a91868997 in term 0.
I20260926 16:41:52.424103 28633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f975b34f31948bdae04d124dd5e27c5" candidate_uuid: "0f75401cf9534a24a1783d9a91868997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9855709147eb45a3bbc578e1424268c9" is_pre_election: true
I20260926 16:41:52.424599 28633 raft_consensus.cc:2454] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f75401cf9534a24a1783d9a91868997 in term 0.
I20260926 16:41:52.424690 28438 leader_election.cc:302] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f75401cf9534a24a1783d9a91868997, b4eeac7eacf94dffbbcf52a12427e313; no voters:
I20260926 16:41:52.425089 28834 raft_consensus.cc:2781] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:41:52.425221 28834 raft_consensus.cc:482] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:41:52.425329 28834 raft_consensus.cc:3037] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:52.427554 28834 raft_consensus.cc:504] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.428210 28834 leader_election.cc:288] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [CANDIDATE]: Term 1 election: Requested vote from peers b4eeac7eacf94dffbbcf52a12427e313 (127.10.110.3:33735), 9855709147eb45a3bbc578e1424268c9 (127.10.110.2:33585)
I20260926 16:41:52.428732 28633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f975b34f31948bdae04d124dd5e27c5" candidate_uuid: "0f75401cf9534a24a1783d9a91868997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9855709147eb45a3bbc578e1424268c9"
I20260926 16:41:52.428732 28763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f975b34f31948bdae04d124dd5e27c5" candidate_uuid: "0f75401cf9534a24a1783d9a91868997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4eeac7eacf94dffbbcf52a12427e313"
I20260926 16:41:52.428941 28763 raft_consensus.cc:3037] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:52.428941 28633 raft_consensus.cc:3037] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:52.432071 28763 raft_consensus.cc:2454] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f75401cf9534a24a1783d9a91868997 in term 1.
I20260926 16:41:52.432094 28633 raft_consensus.cc:2454] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f75401cf9534a24a1783d9a91868997 in term 1.
I20260926 16:41:52.432474 28436 leader_election.cc:302] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f75401cf9534a24a1783d9a91868997, 9855709147eb45a3bbc578e1424268c9; no voters:
I20260926 16:41:52.432783 28834 raft_consensus.cc:2781] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:52.433254 28834 raft_consensus.cc:686] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 1 LEADER]: Becoming Leader. State: Replica: 0f75401cf9534a24a1783d9a91868997, State: Running, Role: LEADER
I20260926 16:41:52.433791 28834 consensus_queue.cc:237] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } }
I20260926 16:41:52.438659 28360 catalog_manager.cc:5495] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f75401cf9534a24a1783d9a91868997 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "0f75401cf9534a24a1783d9a91868997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f75401cf9534a24a1783d9a91868997" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 38523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 } health_report { overall_health: UNKNOWN } } }
I20260926 16:41:52.515777 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.510295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.10.110.1:38523)
0926 16:41:52.510497 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.510502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.510527 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:41:52.510755 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.510759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.510870 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.511009 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:41:52.511364 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.512211 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.513554 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.513957 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.514067 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.514626 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:41:52.515252 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:41:52.515371 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.515470 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.515539 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:41:52.524290 28633 raft_consensus.cc:1261] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 1 FOLLOWER]: Refusing update from remote peer 0f75401cf9534a24a1783d9a91868997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:52.524290 28763 raft_consensus.cc:1261] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 1 FOLLOWER]: Refusing update from remote peer 0f75401cf9534a24a1783d9a91868997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:52.525390 28837 consensus_queue.cc:1035] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4eeac7eacf94dffbbcf52a12427e313" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:52.525756 28834 consensus_queue.cc:1035] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9855709147eb45a3bbc578e1424268c9" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:52.564728 28361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:52.565016 28361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:52.569219 28361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08863e1b78db4e818b2bf72cafb2b380: Sending DeleteTablet for 3 replicas of tablet 0f975b34f31948bdae04d124dd5e27c5
I20260926 16:41:52.570652 28743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f975b34f31948bdae04d124dd5e27c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:52 UTC) from {username='slave'} at 127.0.0.1:47968
I20260926 16:41:52.570657 28613 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f975b34f31948bdae04d124dd5e27c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:52 UTC) from {username='slave'} at 127.0.0.1:53826
I20260926 16:41:52.570978 28482 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f975b34f31948bdae04d124dd5e27c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:52 UTC) from {username='slave'} at 127.0.0.1:40454
I20260926 16:41:52.573055 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28420
I20260926 16:41:52.583787 28851 tablet_replica.cc:330] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: stopping tablet replica
I20260926 16:41:52.584614 28851 raft_consensus.cc:2229] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:41:52.585892 28850 tablet_replica.cc:330] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: stopping tablet replica
I20260926 16:41:52.586094 28849 tablet_replica.cc:330] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: stopping tablet replica
I20260926 16:41:52.586330 28850 raft_consensus.cc:2229] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:52.586447 28849 raft_consensus.cc:2229] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:52.586858 28849 raft_consensus.cc:2258] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:52.586882 28850 raft_consensus.cc:2258] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:52.588088 28851 raft_consensus.cc:2258] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:52.590082 28850 ts_tablet_manager.cc:1905] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:52.590400 28851 ts_tablet_manager.cc:1905] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:52.597833 28851 ts_tablet_manager.cc:1918] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:52.598076 28850 ts_tablet_manager.cc:1918] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:52.598135 28851 log.cc:1196] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/0f975b34f31948bdae04d124dd5e27c5
I20260926 16:41:52.598256 28850 log.cc:1196] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/0f975b34f31948bdae04d124dd5e27c5
I20260926 16:41:52.598857 28851 ts_tablet_manager.cc:1939] T 0f975b34f31948bdae04d124dd5e27c5 P 0f75401cf9534a24a1783d9a91868997: Deleting consensus metadata
I20260926 16:41:52.598867 28850 ts_tablet_manager.cc:1939] T 0f975b34f31948bdae04d124dd5e27c5 P 9855709147eb45a3bbc578e1424268c9: Deleting consensus metadata
I20260926 16:41:52.600914 28346 catalog_manager.cc:4841] TS 9855709147eb45a3bbc578e1424268c9 (127.10.110.2:33585): tablet 0f975b34f31948bdae04d124dd5e27c5 (table test-table [id=e3beef2da9bb4bdea9f93f9f6862e26d]) successfully deleted
I20260926 16:41:52.601476 28348 catalog_manager.cc:4841] TS 0f75401cf9534a24a1783d9a91868997 (127.10.110.1:38523): tablet 0f975b34f31948bdae04d124dd5e27c5 (table test-table [id=e3beef2da9bb4bdea9f93f9f6862e26d]) successfully deleted
I20260926 16:41:52.602479 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.574354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.10.110.1:38523)
0926 16:41:52.574583 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.574590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.585945 (+ 11355us) server_negotiation.cc:407] Connection header received
0926 16:41:52.586262 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.586269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.586373 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.586454 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:52.587235 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.588424 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.590254 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.590843 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.590953 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.600631 (+ 9678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:52.600643 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:52.600662 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:52.601543 (+ 881us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:52.601618 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:52.601658 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:52.601746 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.602016 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.602128 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:41:52.603089 28849 ts_tablet_manager.cc:1905] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:52.603241 28542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:52.611037 28849 ts_tablet_manager.cc:1918] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:52.611323 28849 log.cc:1196] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/0f975b34f31948bdae04d124dd5e27c5
I20260926 16:41:52.612090 28849 ts_tablet_manager.cc:1939] T 0f975b34f31948bdae04d124dd5e27c5 P b4eeac7eacf94dffbbcf52a12427e313: Deleting consensus metadata
I20260926 16:41:52.613945 28348 catalog_manager.cc:4841] TS b4eeac7eacf94dffbbcf52a12427e313 (127.10.110.3:33735): tablet 0f975b34f31948bdae04d124dd5e27c5 (table test-table [id=e3beef2da9bb4bdea9f93f9f6862e26d]) successfully deleted
I20260926 16:41:52.724555 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28420
I20260926 16:41:52.740464 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28551
I20260926 16:41:52.748013 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.741302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.10.110.2:33585)
0926 16:41:52.741516 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.741522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.741600 (+ 78us) server_negotiation.cc:407] Connection header received
0926 16:41:52.741807 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.741812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.741903 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.742001 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:52.742432 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.743320 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.744730 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.745193 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.745313 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.746458 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:52.746468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:52.746483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:52.747119 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:52.747171 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:52.747193 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:52.747309 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.747443 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.747572 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:41:52.748813 28673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:52.839950 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28551
I20260926 16:41:52.854612 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28681
I20260926 16:41:52.861367 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:52.855386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.10.110.3:33735)
0926 16:41:52.855569 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:52.855575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:52.855596 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:41:52.855725 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:52.855730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:52.855851 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:52.855949 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:52.856314 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.857031 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.858422 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:52.858853 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:52.858961 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:52.860044 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:52.860054 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:52.860069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:52.860686 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:52.860738 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:52.860760 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:52.860858 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:52.860984 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 16:41:52.861087 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:41:52.862043 28803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:52.950292 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28681
I20260926 16:41:52.967097 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28330
I20260926 16:41:52.968384 28391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:53.079176 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28330
2026-09-26T16:41:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3106 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:41:53 dist-test-slave-whjp krb5kdc[28859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:41:53 dist-test-slave-whjp krb5kdc[28859](info): set up 2 sockets
Sep 26 16:41:53 dist-test-slave-whjp krb5kdc[28859](info): commencing operation
krb5kdc: starting...
W20260926 16:41:54.116868 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.015s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:41:54 dist-test-slave-whjp krb5kdc[28859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:41:54.369052 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39787
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:54.708463 28872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:54.708783 28872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:54.708911 28872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:54.716826 28872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:41:54.716959 28872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:54.717033 28872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:54.717114 28872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:41:54.717178 28872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:41:54.727808 28872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:39787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:54.729224 28872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:54.730551 28872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:54.742326 28877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:54.742398 28878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:54.742384 28880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:54.743477 28872 server_base.cc:1034] running on GCE node
I20260926 16:41:54.744395 28872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:54.745903 28872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:54.747138 28872 hybrid_clock.cc:648] HybridClock initialized: now 1790440914747077 us; error 51 us; skew 500 ppm
I20260926 16:41:54.747504 28872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:54 dist-test-slave-whjp krb5kdc[28859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440914, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:54.751022 28872 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:41:54.752728 28872 webserver.cc:460] Webserver started at http://127.10.110.62:41571/ using document root <none> and password file <none>
I20260926 16:41:54.753475 28872 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:54.753602 28872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:54.753942 28872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:54.757582 28872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "36af1b4619aa406da7179d6282b1359a"
format_stamp: "Formatted at 2026-09-26 16:41:54 on dist-test-slave-whjp"
server_key: "529454803fea307b61c8ee5112ce3e58"
server_key_iv: "bdfb9a4eaf94613cb0d0c72e4eafaa94"
server_key_version: "encryptionkey@0"
I20260926 16:41:54.758531 28872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "36af1b4619aa406da7179d6282b1359a"
format_stamp: "Formatted at 2026-09-26 16:41:54 on dist-test-slave-whjp"
server_key: "529454803fea307b61c8ee5112ce3e58"
server_key_iv: "bdfb9a4eaf94613cb0d0c72e4eafaa94"
server_key_version: "encryptionkey@0"
I20260926 16:41:54.764843 28872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 16:41:54.770802 28887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:54.772722 28872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:41:54.773139 28872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "36af1b4619aa406da7179d6282b1359a"
format_stamp: "Formatted at 2026-09-26 16:41:54 on dist-test-slave-whjp"
server_key: "529454803fea307b61c8ee5112ce3e58"
server_key_iv: "bdfb9a4eaf94613cb0d0c72e4eafaa94"
server_key_version: "encryptionkey@0"
I20260926 16:41:54.773450 28872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:54.809772 28872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:54.814292 28872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:54.814568 28872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:54.832947 28872 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:39787
I20260926 16:41:54.832953 28938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:39787 every 8 connection(s)
I20260926 16:41:54.834280 28872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:41:54.836375 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28872
I20260926 16:41:54.836545 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:41:54.836987 10680 external_mini_cluster.cc:1428] Setting key 78be7eaa15c01a514be2c47b38e41472
I20260926 16:41:54.838541 28939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:41:54 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:54.852999 28939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a: Bootstrap starting.
I20260926 16:41:54.857223 28939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:54.858484 28939 log.cc:824] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:54.862004 28939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a: No bootstrap required, opened a new log
I20260926 16:41:54.862323 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:54.838790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.10.110.62:39787)
0926 16:41:54.839606 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:54.839623 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:54.839677 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:41:54.841745 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:54.841781 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:54.842335 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:54.842817 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:54.844152 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:54.846343 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:54.848051 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:54.849125 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:54.849324 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:54.853683 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:54.853696 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:54.853717 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:54.853771 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:54.857705 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:54.858493 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:54.858502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:54.858511 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:54.858723 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:54.859234 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:54.859239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:54.859243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:54.859823 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:54.860137 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:54.860579 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 16:41:54.861017 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":319,"threads_started":1}
I20260926 16:41:54.870416 28939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36af1b4619aa406da7179d6282b1359a" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39787 } }
I20260926 16:41:54.870978 28939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:54.871101 28939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36af1b4619aa406da7179d6282b1359a, State: Initialized, Role: FOLLOWER
I20260926 16:41:54.871783 28939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36af1b4619aa406da7179d6282b1359a" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39787 } }
I20260926 16:41:54.872108 28939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:41:54.872255 28939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:41:54.872429 28939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:54.874789 28939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36af1b4619aa406da7179d6282b1359a" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39787 } }
I20260926 16:41:54.875360 28939 leader_election.cc:302] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36af1b4619aa406da7179d6282b1359a; no voters:
I20260926 16:41:54.876117 28939 leader_election.cc:288] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:41:54.876237 28944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:54.876874 28944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [term 1 LEADER]: Becoming Leader. State: Replica: 36af1b4619aa406da7179d6282b1359a, State: Running, Role: LEADER
I20260926 16:41:54.877633 28944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36af1b4619aa406da7179d6282b1359a" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39787 } }
I20260926 16:41:54.878718 28939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:41:54.880805 28946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36af1b4619aa406da7179d6282b1359a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36af1b4619aa406da7179d6282b1359a" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39787 } } }
I20260926 16:41:54.880884 28945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36af1b4619aa406da7179d6282b1359a. Latest consensus state: current_term: 1 leader_uuid: "36af1b4619aa406da7179d6282b1359a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36af1b4619aa406da7179d6282b1359a" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39787 } } }
I20260926 16:41:54.881186 28946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:54.881275 28945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:54.883211 28951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:41:54.889042 28951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:41:54.899300 28951 catalog_manager.cc:1347] Generated new cluster ID: 289dfe0314d946dda5aaf7b9cff0e302
I20260926 16:41:54.899425 28951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:41:54.916742 28951 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:41:54.918843 28951 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:41:54.932469 28951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a: Generated new TSK 0
I20260926 16:41:54.933353 28951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:41:54.999544 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39787
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:55.344883 28965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:55.345221 28965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:55.345347 28965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:55.364590 28965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:55.364714 28965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:55.364862 28965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:41:55.375483 28965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:55.376852 28965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:55.378145 28965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:55.392433 28970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:55.392488 28971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:55.392402 28973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:55.392735 28965 server_base.cc:1034] running on GCE node
I20260926 16:41:55.393802 28965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:55.394830 28965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:55.396063 28965 hybrid_clock.cc:648] HybridClock initialized: now 1790440915395994 us; error 64 us; skew 500 ppm
I20260926 16:41:55.396416 28965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:55 dist-test-slave-whjp krb5kdc[28859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440915, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:55.399776 28965 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:41:55.401374 28965 webserver.cc:460] Webserver started at http://127.10.110.1:42303/ using document root <none> and password file <none>
I20260926 16:41:55.402110 28965 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:55.402240 28965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:55.402572 28965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:55.405746 28965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "ae13d229833445f8a5c11e5116471652"
format_stamp: "Formatted at 2026-09-26 16:41:55 on dist-test-slave-whjp"
server_key: "81331275c16e27b47125eae4bd2cbf93"
server_key_iv: "deda1ebc5abfc48e36b6f5a4ba801c1e"
server_key_version: "encryptionkey@0"
I20260926 16:41:55.406587 28965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "ae13d229833445f8a5c11e5116471652"
format_stamp: "Formatted at 2026-09-26 16:41:55 on dist-test-slave-whjp"
server_key: "81331275c16e27b47125eae4bd2cbf93"
server_key_iv: "deda1ebc5abfc48e36b6f5a4ba801c1e"
server_key_version: "encryptionkey@0"
I20260926 16:41:55.412482 28965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:41:55.416522 28980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:55.418202 28965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:41:55.418397 28965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "ae13d229833445f8a5c11e5116471652"
format_stamp: "Formatted at 2026-09-26 16:41:55 on dist-test-slave-whjp"
server_key: "81331275c16e27b47125eae4bd2cbf93"
server_key_iv: "deda1ebc5abfc48e36b6f5a4ba801c1e"
server_key_version: "encryptionkey@0"
I20260926 16:41:55.418658 28965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:55.439872 28965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:55.445153 28965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:55.445446 28965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:55.446429 28965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:55.447871 28965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:55.447958 28965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:55.448081 28965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:55.448134 28965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:55.480198 28965 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:44827
I20260926 16:41:55.480257 29092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:44827 every 8 connection(s)
I20260926 16:41:55.481491 28965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:41:55.486004 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 28965
I20260926 16:41:55.486173 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:41:55.486601 10680 external_mini_cluster.cc:1428] Setting key ab19385feb440d9e5b0fc0ce970695b9
Sep 26 16:41:55 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440915, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:55.508590 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:55.484034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:35183 (local address 127.10.110.62:39787)
0926 16:41:55.484448 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:55.484452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:55.485651 (+ 1199us) server_negotiation.cc:407] Connection header received
0926 16:41:55.487598 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:55.487604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:55.487702 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:55.487804 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:55.489476 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:55.490202 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:55.491851 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:55.492269 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:55.492386 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:55.502569 (+ 10183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:55.502575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:55.502579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:55.502618 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:55.505275 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:55.506141 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:55.506146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:55.506148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:55.506280 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:55.506745 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:55.506749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:55.506751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:55.507032 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:55.507174 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:55.507979 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 16:41:55.508228 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":208,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
I20260926 16:41:55.509707 29096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:55.484535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39787 (local address 127.10.110.1:35183)
0926 16:41:55.485445 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:55.485496 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:55.487316 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:55.488107 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:55.488117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:55.488675 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:55.489295 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:55.489314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:55.490376 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:55.490380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:55.491666 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:55.491682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:55.492512 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:55.492519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:55.492763 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:55.499203 (+ 6440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:55.499259 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:55.502259 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:55.505478 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:55.505487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:55.505503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:55.505992 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:55.506396 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:55.506401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:55.506405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:55.506621 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:55.507177 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:55.507186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:55.507560 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 16:41:55.507808 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 16:41:55.508411 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":313,"threads_started":1}
I20260926 16:41:55.511479 29093 heartbeater.cc:347] Connected to a master server at 127.10.110.62:39787
I20260926 16:41:55.511790 29093 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
I20260926 16:41:55.512619 29093 heartbeater.cc:511] Master 127.10.110.62:39787 requested a full tablet report, sending...
I20260926 16:41:55.514880 28904 ts_manager.cc:194] Registered new tserver with Master: ae13d229833445f8a5c11e5116471652 (127.10.110.1:44827)
I20260926 16:41:55.516623 28904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:35183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:41:55.545234 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39787
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:55.893788 29099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:55.894109 29099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:55.894232 29099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:55.903237 29099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:55.903400 29099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:55.903555 29099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:41:55.914264 29099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:55.915643 29099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:55.917050 29099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:55.930996 29104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:55.930974 29105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:55.931082 29107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:55.931864 29099 server_base.cc:1034] running on GCE node
I20260926 16:41:55.932570 29099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:55.933570 29099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:55.934815 29099 hybrid_clock.cc:648] HybridClock initialized: now 1790440915934750 us; error 64 us; skew 500 ppm
I20260926 16:41:55.935197 29099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:55 dist-test-slave-whjp krb5kdc[28859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440915, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:55.938482 29099 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:41:55.940352 29099 webserver.cc:460] Webserver started at http://127.10.110.2:42111/ using document root <none> and password file <none>
I20260926 16:41:55.941051 29099 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:55.941162 29099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:55.941437 29099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:55.944501 29099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "6db1f28ac02846759d683920697a0ee0"
format_stamp: "Formatted at 2026-09-26 16:41:55 on dist-test-slave-whjp"
server_key: "7a3fd4c51acf57b445f1258b54e2de5a"
server_key_iv: "7c3b1c4263558fb55cdda56ff86604bb"
server_key_version: "encryptionkey@0"
I20260926 16:41:55.945281 29099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "6db1f28ac02846759d683920697a0ee0"
format_stamp: "Formatted at 2026-09-26 16:41:55 on dist-test-slave-whjp"
server_key: "7a3fd4c51acf57b445f1258b54e2de5a"
server_key_iv: "7c3b1c4263558fb55cdda56ff86604bb"
server_key_version: "encryptionkey@0"
I20260926 16:41:55.951261 29099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:41:55.955360 29114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:55.957121 29099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:41:55.957301 29099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "6db1f28ac02846759d683920697a0ee0"
format_stamp: "Formatted at 2026-09-26 16:41:55 on dist-test-slave-whjp"
server_key: "7a3fd4c51acf57b445f1258b54e2de5a"
server_key_iv: "7c3b1c4263558fb55cdda56ff86604bb"
server_key_version: "encryptionkey@0"
I20260926 16:41:55.957494 29099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:55.971128 29099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:55.975977 29099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:55.976243 29099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:55.977275 29099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:55.978821 29099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:55.978907 29099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:55.979017 29099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:55.979085 29099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:56.018981 29099 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:33975
I20260926 16:41:56.018981 29226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:33975 every 8 connection(s)
I20260926 16:41:56.021144 29099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:41:56.021471 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29099
I20260926 16:41:56.021664 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:41:56.022382 10680 external_mini_cluster.cc:1428] Setting key 5015feef30e57d9e6fdb0fa17ec8f470
Sep 26 16:41:56 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440915, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:56.049707 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.024531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:33225 (local address 127.10.110.62:39787)
0926 16:41:56.024989 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.024995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.026467 (+ 1472us) server_negotiation.cc:407] Connection header received
0926 16:41:56.029269 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.029275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.029453 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.029687 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:56.032258 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.033229 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.035903 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.036732 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.036980 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.041727 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:56.041735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:56.041740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:56.041792 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:56.045369 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.046380 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.046386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.046390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.046579 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.047277 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.047283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.047285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.047668 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:56.047869 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.048930 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.049292 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260926 16:41:56.050420 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.025144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39787 (local address 127.10.110.2:33225)
0926 16:41:56.026213 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.026280 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.028744 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.030022 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.030036 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.030911 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:56.031949 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.031996 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.033457 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.033463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.035599 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.035629 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.037110 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.037117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.037357 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.038309 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:56.038336 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:56.041372 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:56.045610 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:56.045621 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:56.045639 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:56.046214 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:56.046808 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:56.046812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:56.046815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:56.047127 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:56.047879 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:56.047890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:56.048488 (+ 598us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.048762 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.049218 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":340,"threads_started":1}
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
I20260926 16:41:56.052192 29227 heartbeater.cc:347] Connected to a master server at 127.10.110.62:39787
I20260926 16:41:56.052511 29227 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:56.053344 29227 heartbeater.cc:511] Master 127.10.110.62:39787 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
I20260926 16:41:56.055073 28904 ts_manager.cc:194] Registered new tserver with Master: 6db1f28ac02846759d683920697a0ee0 (127.10.110.2:33975)
I20260926 16:41:56.055991 28904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:33225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:41:56.088372 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39787
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:41:56.447396 29233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:56.447799 29233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:56.447976 29233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:56.456714 29233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:56.456856 29233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:56.457013 29233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:41:56.468999 29233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:56.472277 29233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:56.474361 29233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:56.492836 29238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:56.492986 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:56.493006 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:56.493736 29233 server_base.cc:1034] running on GCE node
I20260926 16:41:56.494565 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:56.495774 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:56.497186 29233 hybrid_clock.cc:648] HybridClock initialized: now 1790440916497025 us; error 49 us; skew 500 ppm
I20260926 16:41:56.497717 29233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:41:56 dist-test-slave-whjp krb5kdc[28859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440916, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:41:56.501531 29233 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:41:56.503328 29233 webserver.cc:460] Webserver started at http://127.10.110.3:45115/ using document root <none> and password file <none>
I20260926 16:41:56.504130 29233 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:56.504251 29233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:56.504549 29233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:56.508684 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "705a9edde6c04256ace9989c99ee2e17"
format_stamp: "Formatted at 2026-09-26 16:41:56 on dist-test-slave-whjp"
server_key: "35011b0cb39726f9852ec939a179f51f"
server_key_iv: "7996f08ae2fd2f751d0b6524f063951a"
server_key_version: "encryptionkey@0"
I20260926 16:41:56.509542 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "705a9edde6c04256ace9989c99ee2e17"
format_stamp: "Formatted at 2026-09-26 16:41:56 on dist-test-slave-whjp"
server_key: "35011b0cb39726f9852ec939a179f51f"
server_key_iv: "7996f08ae2fd2f751d0b6524f063951a"
server_key_version: "encryptionkey@0"
I20260926 16:41:56.515707 29233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260926 16:41:56.520021 29248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:56.520625 29093 heartbeater.cc:503] Master 127.10.110.62:39787 was elected leader, sending a full tablet report...
I20260926 16:41:56.522208 29233 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:41:56.522454 29233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "705a9edde6c04256ace9989c99ee2e17"
format_stamp: "Formatted at 2026-09-26 16:41:56 on dist-test-slave-whjp"
server_key: "35011b0cb39726f9852ec939a179f51f"
server_key_iv: "7996f08ae2fd2f751d0b6524f063951a"
server_key_version: "encryptionkey@0"
I20260926 16:41:56.522774 29233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:56.555949 29233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:56.561460 29233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:56.561741 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:56.562889 29233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:56.564534 29233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:56.564627 29233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:56.564719 29233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:56.564770 29233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:56.603088 29233 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:39735
I20260926 16:41:56.603176 29360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:39735 every 8 connection(s)
I20260926 16:41:56.604489 29233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:41:56.614677 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29233
I20260926 16:41:56.614880 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:41:56.615468 10680 external_mini_cluster.cc:1428] Setting key 1f2b312699bd0cd3af04e3138b53df35
Sep 26 16:41:56 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440916, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:41:56.627724 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.607213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:35481 (local address 127.10.110.62:39787)
0926 16:41:56.607725 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.607730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.608842 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 16:41:56.610911 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.610917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.611005 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.611115 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:56.612858 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.613680 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.616064 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.616605 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.616726 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.620509 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:56.620514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:56.620518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:56.620557 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:56.623730 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.624878 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.624884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.624886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.625053 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.625575 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.625579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.625583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.625894 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:56.626055 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.626926 (+ 871us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.627211 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260926 16:41:56.628820 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.607808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39787 (local address 127.10.110.3:35481)
0926 16:41:56.608634 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.608682 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.610533 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.611398 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.611407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.611997 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:56.612646 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.612666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.613849 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.613853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.615809 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.615861 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.616845 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.616853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.617134 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.617813 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:41:56.617836 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:56.620229 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:56.623985 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:56.623996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:56.624106 (+ 110us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:56.624692 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:56.625195 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:41:56.625201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:41:56.625205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:41:56.625443 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:41:56.626044 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:56.626053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:56.626446 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.626775 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.627444 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":288,"threads_started":1}
I20260926 16:41:56.630378 29361 heartbeater.cc:347] Connected to a master server at 127.10.110.62:39787
I20260926 16:41:56.630743 29361 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:56.631669 29361 heartbeater.cc:511] Master 127.10.110.62:39787 requested a full tablet report, sending...
I20260926 16:41:56.633826 28903 ts_manager.cc:194] Registered new tserver with Master: 705a9edde6c04256ace9989c99ee2e17 (127.10.110.3:39735)
I20260926 16:41:56.634775 28903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:35481
I20260926 16:41:56.644662 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:41:56.664987 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.649213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.10.110.62:39787)
0926 16:41:56.649430 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.649438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.649860 (+ 422us) server_negotiation.cc:407] Connection header received
0926 16:41:56.650064 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.650069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.650144 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.650251 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:56.651791 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.652688 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.654232 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.654723 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.654847 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.657834 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:56.657840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:56.657843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:56.657887 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:56.661021 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.662421 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.662426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.662428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.662601 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.663317 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.663322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.663324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.663692 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:56.663982 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.664397 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.664651 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:41:56.669996 28903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:41:56.677732 28903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:41:56.720070 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.693159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33975 (local address 127.0.0.1:54808)
0926 16:41:56.695114 (+ 1955us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.695168 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.695366 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.710751 (+ 15385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.710762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.710811 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:56.711367 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.711390 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.713770 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.713776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.716438 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.716461 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.718927 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.718949 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.719206 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.719269 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.719415 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.719626 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1676,"thread_start_us":20127,"threads_started":1}
I20260926 16:41:56.720453 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.693557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.10.110.2:33975)
0926 16:41:56.709518 (+ 15961us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.709530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.709576 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:41:56.709725 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.709736 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.710069 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.711710 (+ 1641us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:56.711996 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.713626 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.716669 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.718339 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.718515 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.718752 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.719893 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.720100 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15727,"thread_start_us":16788,"threads_started":1}
I20260926 16:41:56.721585 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.695569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:39735 (local address 127.0.0.1:41658)
0926 16:41:56.713049 (+ 17480us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.713068 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.713204 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.714291 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.714296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.714331 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:56.714923 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.714941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.716880 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.716884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.718432 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.718450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.720663 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.720670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.720848 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.720885 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.720979 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.721126 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17359,"mutex_wait_us":1,"thread_start_us":17205,"threads_started":1}
I20260926 16:41:56.721812 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.694073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.10.110.1:44827)
0926 16:41:56.711325 (+ 17252us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.711334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.713347 (+ 2013us) server_negotiation.cc:407] Connection header received
0926 16:41:56.713442 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.713448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.713652 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.713886 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:56.714801 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.716115 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.718867 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.720379 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.720521 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.720760 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.721314 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.721495 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17053,"thread_start_us":16898,"threads_started":1}
I20260926 16:41:56.722389 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.695982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.10.110.3:39735)
0926 16:41:56.705903 (+ 9921us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.705914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.713196 (+ 7282us) server_negotiation.cc:407] Connection header received
0926 16:41:56.713353 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.713361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.713700 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.713886 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:56.715172 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.716735 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.718655 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.720130 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.720272 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.720505 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.721925 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.722108 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9701,"thread_start_us":9499,"threads_started":1}
I20260926 16:41:56.725129 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.693524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:44827 (local address 127.0.0.1:51496)
0926 16:41:56.712230 (+ 18706us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.712264 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.712486 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.713875 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.713880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.713910 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:56.714510 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.714524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.716272 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.716279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.718657 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.718672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.720918 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.720925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.721104 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.721137 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.721220 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.721362 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18348,"thread_start_us":1187,"threads_started":1}
I20260926 16:41:56.724834 29162 tablet_service.cc:1467] Processing CreateTablet for tablet d9a9571c31f344ee8923fe61d61dd4fb (DEFAULT_TABLE table=test-table [id=27f47c1dc69b452098c28d0d22ef22d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:56.726450 29162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a9571c31f344ee8923fe61d61dd4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:56.729141 29296 tablet_service.cc:1467] Processing CreateTablet for tablet d9a9571c31f344ee8923fe61d61dd4fb (DEFAULT_TABLE table=test-table [id=27f47c1dc69b452098c28d0d22ef22d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:56.729655 29028 tablet_service.cc:1467] Processing CreateTablet for tablet d9a9571c31f344ee8923fe61d61dd4fb (DEFAULT_TABLE table=test-table [id=27f47c1dc69b452098c28d0d22ef22d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:56.730568 29296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a9571c31f344ee8923fe61d61dd4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:56.731464 29028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a9571c31f344ee8923fe61d61dd4fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:56.743149 29380 tablet_bootstrap.cc:492] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Bootstrap starting.
I20260926 16:41:56.747148 29381 tablet_bootstrap.cc:492] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Bootstrap starting.
I20260926 16:41:56.748670 29380 tablet_bootstrap.cc:654] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:56.750124 29380 log.cc:824] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:56.751159 29382 tablet_bootstrap.cc:492] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Bootstrap starting.
I20260926 16:41:56.751792 29381 tablet_bootstrap.cc:654] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:56.753381 29381 log.cc:824] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:56.755220 29382 tablet_bootstrap.cc:654] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:56.755606 29380 tablet_bootstrap.cc:492] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: No bootstrap required, opened a new log
I20260926 16:41:56.755972 29380 ts_tablet_manager.cc:1397] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260926 16:41:56.756158 29381 tablet_bootstrap.cc:492] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: No bootstrap required, opened a new log
I20260926 16:41:56.756426 29381 ts_tablet_manager.cc:1397] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:41:56.756654 29382 log.cc:824] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:56.760293 29382 tablet_bootstrap.cc:492] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: No bootstrap required, opened a new log
I20260926 16:41:56.760633 29382 ts_tablet_manager.cc:1397] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260926 16:41:56.765018 29381 raft_consensus.cc:348] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.765450 29381 raft_consensus.cc:374] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:56.765573 29381 raft_consensus.cc:729] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705a9edde6c04256ace9989c99ee2e17, State: Initialized, Role: FOLLOWER
I20260926 16:41:56.766227 29381 consensus_queue.cc:260] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.766600 29380 raft_consensus.cc:348] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.767175 29380 raft_consensus.cc:374] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:56.767290 29380 raft_consensus.cc:729] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db1f28ac02846759d683920697a0ee0, State: Initialized, Role: FOLLOWER
I20260926 16:41:56.767419 29361 heartbeater.cc:503] Master 127.10.110.62:39787 was elected leader, sending a full tablet report...
I20260926 16:41:56.768095 29381 ts_tablet_manager.cc:1428] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260926 16:41:56.767999 29380 consensus_queue.cc:260] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.769194 29227 heartbeater.cc:503] Master 127.10.110.62:39787 was elected leader, sending a full tablet report...
I20260926 16:41:56.770040 29380 ts_tablet_manager.cc:1428] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 16:41:56.773723 29228 tablet.cc:2369] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:56.773963 29382 raft_consensus.cc:348] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.774533 29382 raft_consensus.cc:374] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:56.774652 29382 raft_consensus.cc:729] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae13d229833445f8a5c11e5116471652, State: Initialized, Role: FOLLOWER
I20260926 16:41:56.775292 29382 consensus_queue.cc:260] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.777359 29382 ts_tablet_manager.cc:1428] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Time spent starting tablet: real 0.017s user 0.011s sys 0.002s
I20260926 16:41:56.827626 29386 raft_consensus.cc:482] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:41:56.827987 29386 raft_consensus.cc:504] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.829790 29386 leader_election.cc:288] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6db1f28ac02846759d683920697a0ee0 (127.10.110.2:33975), ae13d229833445f8a5c11e5116471652 (127.10.110.1:44827)
I20260926 16:41:56.836935 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.830329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:53429 (local address 127.10.110.2:33975)
0926 16:41:56.830507 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.830513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.830533 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:41:56.830793 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.830799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.830882 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.831000 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:56.831521 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.832522 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.834650 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.835941 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.836111 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.836191 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.836564 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.836703 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:41:56.836926 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.830115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33975 (local address 127.10.110.3:53429)
0926 16:41:56.830447 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.830471 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.830635 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.831026 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.831029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.831053 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:56.831322 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.831335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.832665 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.832670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.834444 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.834461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.836120 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.836126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.836314 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.836353 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.836452 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.836619 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"spinlock_wait_cycles":35072}
I20260926 16:41:56.837257 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.830262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:43485 (local address 127.10.110.1:44827)
0926 16:41:56.830465 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.830472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.831112 (+ 640us) server_negotiation.cc:407] Connection header received
0926 16:41:56.831292 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.831298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.831378 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.831465 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:56.832174 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.833197 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.835303 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.836227 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.836395 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.836469 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.836851 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.837005 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:41:56.838023 29389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.830180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:44827 (local address 127.10.110.3:43485)
0926 16:41:56.830982 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:56.831005 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:56.831151 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:56.831521 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:56.831526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:56.831553 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:56.832000 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.832014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.833373 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.833378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.835134 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:56.835146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.836443 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.836449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:56.836643 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.836679 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:41:56.837460 (+ 781us) client_negotiation.cc:240] Negotiation successful
0926 16:41:56.837623 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":1,"thread_start_us":450,"threads_started":1}
I20260926 16:41:56.838533 29182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a9571c31f344ee8923fe61d61dd4fb" candidate_uuid: "705a9edde6c04256ace9989c99ee2e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db1f28ac02846759d683920697a0ee0" is_pre_election: true
I20260926 16:41:56.839103 29182 raft_consensus.cc:2454] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 705a9edde6c04256ace9989c99ee2e17 in term 0.
I20260926 16:41:56.839140 29048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a9571c31f344ee8923fe61d61dd4fb" candidate_uuid: "705a9edde6c04256ace9989c99ee2e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae13d229833445f8a5c11e5116471652" is_pre_election: true
I20260926 16:41:56.839665 29048 raft_consensus.cc:2454] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 705a9edde6c04256ace9989c99ee2e17 in term 0.
I20260926 16:41:56.839924 29250 leader_election.cc:302] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db1f28ac02846759d683920697a0ee0, 705a9edde6c04256ace9989c99ee2e17; no voters:
I20260926 16:41:56.840349 29386 raft_consensus.cc:2781] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:41:56.840510 29386 raft_consensus.cc:482] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:41:56.840622 29386 raft_consensus.cc:3037] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:56.842871 29386 raft_consensus.cc:504] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.843636 29386 leader_election.cc:288] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [CANDIDATE]: Term 1 election: Requested vote from peers 6db1f28ac02846759d683920697a0ee0 (127.10.110.2:33975), ae13d229833445f8a5c11e5116471652 (127.10.110.1:44827)
I20260926 16:41:56.844215 29182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a9571c31f344ee8923fe61d61dd4fb" candidate_uuid: "705a9edde6c04256ace9989c99ee2e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db1f28ac02846759d683920697a0ee0"
I20260926 16:41:56.844301 29048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a9571c31f344ee8923fe61d61dd4fb" candidate_uuid: "705a9edde6c04256ace9989c99ee2e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae13d229833445f8a5c11e5116471652"
I20260926 16:41:56.844472 29182 raft_consensus.cc:3037] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:56.844513 29048 raft_consensus.cc:3037] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:56.847856 29048 raft_consensus.cc:2454] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705a9edde6c04256ace9989c99ee2e17 in term 1.
I20260926 16:41:56.847854 29182 raft_consensus.cc:2454] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705a9edde6c04256ace9989c99ee2e17 in term 1.
I20260926 16:41:56.848467 29252 leader_election.cc:302] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6db1f28ac02846759d683920697a0ee0, 705a9edde6c04256ace9989c99ee2e17, ae13d229833445f8a5c11e5116471652; no voters:
I20260926 16:41:56.848905 29386 raft_consensus.cc:2781] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:56.849354 29386 raft_consensus.cc:686] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 1 LEADER]: Becoming Leader. State: Replica: 705a9edde6c04256ace9989c99ee2e17, State: Running, Role: LEADER
I20260926 16:41:56.849915 29386 consensus_queue.cc:237] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } }
I20260926 16:41:56.854974 28904 catalog_manager.cc:5495] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 705a9edde6c04256ace9989c99ee2e17 (127.10.110.3). New cstate: current_term: 1 leader_uuid: "705a9edde6c04256ace9989c99ee2e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705a9edde6c04256ace9989c99ee2e17" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 39735 } health_report { overall_health: HEALTHY } } }
W20260926 16:41:56.856127 29362 tablet.cc:2369] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:56.905508 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.899816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.10.110.3:39735)
0926 16:41:56.900033 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.900038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.900058 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:41:56.900169 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.900173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.900274 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.900411 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:41:56.900755 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.901473 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.903005 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.903482 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.903586 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.904258 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:41:56.904789 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:41:56.904928 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.905173 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.905241 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:41:56.914780 29182 raft_consensus.cc:1261] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 1 FOLLOWER]: Refusing update from remote peer 705a9edde6c04256ace9989c99ee2e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:56.914780 29048 raft_consensus.cc:1261] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 1 FOLLOWER]: Refusing update from remote peer 705a9edde6c04256ace9989c99ee2e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:41:56.916021 29390 consensus_queue.cc:1035] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db1f28ac02846759d683920697a0ee0" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:56.916554 29386 consensus_queue.cc:1035] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae13d229833445f8a5c11e5116471652" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:41:56.930714 29393 mvcc.cc:205] Tried to move back new op lower bound from 7333645995671879680 to 7333645995421507584. Current Snapshot: MvccSnapshot[applied={T|T < 7333645995671879680}]
I20260926 16:41:56.949510 28903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:56.949971 28903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:41:56.954804 28903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36af1b4619aa406da7179d6282b1359a: Sending DeleteTablet for 3 replicas of tablet d9a9571c31f344ee8923fe61d61dd4fb
I20260926 16:41:56.956436 29162 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a9571c31f344ee8923fe61d61dd4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:56 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:54808
I20260926 16:41:56.956751 29296 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a9571c31f344ee8923fe61d61dd4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:56 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:41658
I20260926 16:41:56.957821 29028 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a9571c31f344ee8923fe61d61dd4fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:41:56 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260926 16:41:56.958385 29403 tablet_replica.cc:330] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: stopping tablet replica
I20260926 16:41:56.958549 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28965
I20260926 16:41:56.958945 29403 raft_consensus.cc:2229] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:41:56.959672 29403 raft_consensus.cc:2258] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:56.963630 29404 tablet_replica.cc:330] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: stopping tablet replica
I20260926 16:41:56.964318 29404 raft_consensus.cc:2229] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:56.965873 29404 raft_consensus.cc:2258] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:56.968467 29403 ts_tablet_manager.cc:1905] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:56.968747 29404 ts_tablet_manager.cc:1905] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:56.972211 29402 tablet_replica.cc:330] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: stopping tablet replica
I20260926 16:41:56.972889 29402 raft_consensus.cc:2229] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:41:56.974004 29402 raft_consensus.cc:2258] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:41:56.977593 29404 ts_tablet_manager.cc:1918] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:56.977982 29404 log.cc:1196] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/d9a9571c31f344ee8923fe61d61dd4fb
I20260926 16:41:56.978570 29403 ts_tablet_manager.cc:1918] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:56.978704 29404 ts_tablet_manager.cc:1939] T d9a9571c31f344ee8923fe61d61dd4fb P ae13d229833445f8a5c11e5116471652: Deleting consensus metadata
Sep 26 16:41:56 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:41:56.979048 29403 log.cc:1196] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/d9a9571c31f344ee8923fe61d61dd4fb
I20260926 16:41:56.979822 29403 ts_tablet_manager.cc:1939] T d9a9571c31f344ee8923fe61d61dd4fb P 705a9edde6c04256ace9989c99ee2e17: Deleting consensus metadata
I20260926 16:41:56.981158 29402 ts_tablet_manager.cc:1905] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:41:56.981812 28891 catalog_manager.cc:4841] TS ae13d229833445f8a5c11e5116471652 (127.10.110.1:44827): tablet d9a9571c31f344ee8923fe61d61dd4fb (table test-table [id=27f47c1dc69b452098c28d0d22ef22d6]) successfully deleted
I20260926 16:41:56.982199 28891 catalog_manager.cc:4841] TS 705a9edde6c04256ace9989c99ee2e17 (127.10.110.3:39735): tablet d9a9571c31f344ee8923fe61d61dd4fb (table test-table [id=27f47c1dc69b452098c28d0d22ef22d6]) successfully deleted
I20260926 16:41:56.988674 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:56.960883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.10.110.1:44827)
0926 16:41:56.961099 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:56.961105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:56.963224 (+ 2119us) server_negotiation.cc:407] Connection header received
0926 16:41:56.963637 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:56.963643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:56.963729 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:56.963807 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:56.965775 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.967126 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.969277 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:56.970278 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:56.970416 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:56.980529 (+ 10113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:56.980551 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:56.980571 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:56.980623 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:56.984599 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.985705 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.985714 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.985717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.985894 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:56.986480 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:56.986484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:56.986486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:56.986962 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:56.987460 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:56.987904 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 16:41:56.988203 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:41:56.989414 29088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:56.989791 29402 ts_tablet_manager.cc:1918] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:41:56.990206 29402 log.cc:1196] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/d9a9571c31f344ee8923fe61d61dd4fb
I20260926 16:41:56.991420 29402 ts_tablet_manager.cc:1939] T d9a9571c31f344ee8923fe61d61dd4fb P 6db1f28ac02846759d683920697a0ee0: Deleting consensus metadata
I20260926 16:41:56.995133 28889 catalog_manager.cc:4841] TS 6db1f28ac02846759d683920697a0ee0 (127.10.110.2:33975): tablet d9a9571c31f344ee8923fe61d61dd4fb (table test-table [id=27f47c1dc69b452098c28d0d22ef22d6]) successfully deleted
I20260926 16:41:57.090245 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28965
I20260926 16:41:57.106550 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29099
Sep 26 16:41:57 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:41:57.124231 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:57.107681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.10.110.2:33975)
0926 16:41:57.107882 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:57.107887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:57.107905 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:41:57.107972 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:57.107977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:57.108067 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:57.108171 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:57.109466 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:57.110261 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:57.111639 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:57.112211 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:57.112333 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:57.117279 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:57.117297 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:57.117314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:57.117359 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:57.120725 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:57.121939 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:57.121951 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:57.121955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:57.122125 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:57.122645 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:57.122650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:57.122652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:57.123089 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:57.123335 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:57.123692 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 16:41:57.123944 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 16:41:57.124979 29222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:57.224501 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29099
I20260926 16:41:57.246850 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29233
Sep 26 16:41:57 dist-test-slave-whjp krb5kdc[28859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:41:57.267953 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:57.247740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.10.110.3:39735)
0926 16:41:57.247953 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:57.247958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:57.247978 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:41:57.248165 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:57.248169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:57.248241 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:57.248337 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:57.250326 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:57.251233 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:57.253657 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:57.254210 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:57.254335 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:57.260499 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:57.260518 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:57.260586 (+ 68us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:41:57.260633 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:57.264131 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:57.265290 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:57.265300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:57.265302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:57.265584 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:41:57.266097 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:41:57.266101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:41:57.266103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:41:57.266494 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:57.266812 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:57.267267 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 16:41:57.267508 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:41:57.268867 29356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:57.395916 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29233
I20260926 16:41:57.412111 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 28872
I20260926 16:41:57.413275 28934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:41:57.527814 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 28872
2026-09-26T16:41:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4450 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T16:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:41:57Z Disabled control of system clock
I20260926 16:41:57.602134 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44923
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:41:57.979625 29413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:57.980016 29413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:57.980116 29413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:57.988667 29413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:41:57.988826 29413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:57.988869 29413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:57.988945 29413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:41:57.988998 29413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:41:58.000388 29413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:44923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:58.001823 29413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:58.003793 29413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:58.016990 29421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:58.016990 29419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:58.017764 29418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:58.018136 29413 server_base.cc:1034] running on GCE node
I20260926 16:41:58.018996 29413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:58.020692 29413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:58.021921 29413 hybrid_clock.cc:648] HybridClock initialized: now 1790440918021866 us; error 56 us; skew 500 ppm
I20260926 16:41:58.022310 29413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:58.024912 29413 webserver.cc:460] Webserver started at http://127.10.110.62:45379/ using document root <none> and password file <none>
I20260926 16:41:58.025731 29413 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:58.025854 29413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:58.026221 29413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:58.029695 29413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "4095c09c783e4116aa05b5a5e413c052"
format_stamp: "Formatted at 2026-09-26 16:41:58 on dist-test-slave-whjp"
server_key: "aa7f3ee2651a775ca6c781379392a18b"
server_key_iv: "fc8d97cdd3d34f903696ce00e2075005"
server_key_version: "encryptionkey@0"
I20260926 16:41:58.030975 29413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "4095c09c783e4116aa05b5a5e413c052"
format_stamp: "Formatted at 2026-09-26 16:41:58 on dist-test-slave-whjp"
server_key: "aa7f3ee2651a775ca6c781379392a18b"
server_key_iv: "fc8d97cdd3d34f903696ce00e2075005"
server_key_version: "encryptionkey@0"
I20260926 16:41:58.037395 29413 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:41:58.041818 29427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:58.043478 29413 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 16:41:58.043702 29413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "4095c09c783e4116aa05b5a5e413c052"
format_stamp: "Formatted at 2026-09-26 16:41:58 on dist-test-slave-whjp"
server_key: "aa7f3ee2651a775ca6c781379392a18b"
server_key_iv: "fc8d97cdd3d34f903696ce00e2075005"
server_key_version: "encryptionkey@0"
I20260926 16:41:58.043972 29413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:58.057664 29413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:58.058624 29413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:58.058928 29413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:58.078608 29413 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:44923
I20260926 16:41:58.078686 29478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:44923 every 8 connection(s)
I20260926 16:41:58.080130 29413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:41:58.083695 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29413
I20260926 16:41:58.083911 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:41:58.084455 10680 external_mini_cluster.cc:1428] Setting key 805514c84f305d768cedab1db9b88ba1
I20260926 16:41:58.084720 29479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:58.100819 29479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052: Bootstrap starting.
I20260926 16:41:58.101281 29482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:58.086596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.10.110.62:44923)
0926 16:41:58.088321 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:58.088339 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:58.088406 (+ 67us) server_negotiation.cc:407] Connection header received
0926 16:41:58.090243 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:58.090275 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:58.090873 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:58.091290 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:58.091981 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:58.093411 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:58.095516 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:58.096504 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:58.096710 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:58.098377 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:58.098387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:58.098409 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:58.099053 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:58.099160 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:58.099194 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:58.099309 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:58.099633 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 16:41:58.100115 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":527,"threads_started":1}
I20260926 16:41:58.104701 29479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:58.105855 29479 log.cc:824] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:58.109184 29479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052: No bootstrap required, opened a new log
I20260926 16:41:58.119590 29479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095c09c783e4116aa05b5a5e413c052" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44923 } }
I20260926 16:41:58.120224 29479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:58.120357 29479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4095c09c783e4116aa05b5a5e413c052, State: Initialized, Role: FOLLOWER
I20260926 16:41:58.121006 29479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095c09c783e4116aa05b5a5e413c052" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44923 } }
I20260926 16:41:58.121273 29479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:41:58.121428 29479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:41:58.121583 29479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:58.124105 29479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095c09c783e4116aa05b5a5e413c052" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44923 } }
I20260926 16:41:58.124845 29479 leader_election.cc:302] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4095c09c783e4116aa05b5a5e413c052; no voters:
I20260926 16:41:58.125566 29479 leader_election.cc:288] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:41:58.125766 29484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:58.126487 29484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [term 1 LEADER]: Becoming Leader. State: Replica: 4095c09c783e4116aa05b5a5e413c052, State: Running, Role: LEADER
I20260926 16:41:58.127089 29484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095c09c783e4116aa05b5a5e413c052" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44923 } }
I20260926 16:41:58.128400 29479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:41:58.130447 29485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4095c09c783e4116aa05b5a5e413c052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095c09c783e4116aa05b5a5e413c052" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44923 } } }
I20260926 16:41:58.130997 29485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:58.132055 29492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:41:58.131791 29486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4095c09c783e4116aa05b5a5e413c052. Latest consensus state: current_term: 1 leader_uuid: "4095c09c783e4116aa05b5a5e413c052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4095c09c783e4116aa05b5a5e413c052" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 44923 } } }
I20260926 16:41:58.132341 29486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052 [sys.catalog]: This master's current role is: LEADER
I20260926 16:41:58.139351 29492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:41:58.150851 29492 catalog_manager.cc:1347] Generated new cluster ID: acd5693c4f754dbbadfdb2d3f07e2ed9
I20260926 16:41:58.151003 29492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:41:58.167021 29492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:41:58.169142 29492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:41:58.183053 29492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052: Generated new TSK 0
I20260926 16:41:58.184113 29492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:41:58.197656 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44923
--builtin_ntp_servers=127.10.110.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:41:58.583387 29503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:58.583734 29503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:58.583911 29503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:58.591811 29503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:58.591996 29503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:58.592149 29503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:41:58.601958 29503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:58.603403 29503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:58.605471 29503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:58.623473 29508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:58.623747 29509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:58.623751 29511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:58.625360 29503 server_base.cc:1034] running on GCE node
I20260926 16:41:58.626127 29503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:58.627318 29503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:58.628912 29503 hybrid_clock.cc:648] HybridClock initialized: now 1790440918628811 us; error 64 us; skew 500 ppm
I20260926 16:41:58.630489 29503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:58.633505 29503 webserver.cc:460] Webserver started at http://127.10.110.1:33705/ using document root <none> and password file <none>
I20260926 16:41:58.634526 29503 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:58.634675 29503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:58.635078 29503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:58.638427 29503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "e542e6d3cb6746e680bf5953c8dbc7ba"
format_stamp: "Formatted at 2026-09-26 16:41:58 on dist-test-slave-whjp"
server_key: "7f3ca17de2b850b0fb2e2ad6140cf066"
server_key_iv: "1df6d326455db5352ff65f8d15b229c7"
server_key_version: "encryptionkey@0"
I20260926 16:41:58.639369 29503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "e542e6d3cb6746e680bf5953c8dbc7ba"
format_stamp: "Formatted at 2026-09-26 16:41:58 on dist-test-slave-whjp"
server_key: "7f3ca17de2b850b0fb2e2ad6140cf066"
server_key_iv: "1df6d326455db5352ff65f8d15b229c7"
server_key_version: "encryptionkey@0"
I20260926 16:41:58.645726 29503 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:41:58.650439 29517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:58.652541 29503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:41:58.652803 29503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "e542e6d3cb6746e680bf5953c8dbc7ba"
format_stamp: "Formatted at 2026-09-26 16:41:58 on dist-test-slave-whjp"
server_key: "7f3ca17de2b850b0fb2e2ad6140cf066"
server_key_iv: "1df6d326455db5352ff65f8d15b229c7"
server_key_version: "encryptionkey@0"
I20260926 16:41:58.653272 29503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:58.670915 29503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:58.671964 29503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:58.672313 29503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:58.673601 29503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:58.675879 29503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:58.676013 29503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:41:58.676144 29503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:58.676224 29503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:58.709568 29503 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:42115
I20260926 16:41:58.709610 29629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:42115 every 8 connection(s)
I20260926 16:41:58.710971 29503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:41:58.716045 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29503
I20260926 16:41:58.716223 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:41:58.716647 10680 external_mini_cluster.cc:1428] Setting key 55168b57c8927a9ad10400fc3e26da4c
I20260926 16:41:58.719769 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44923
--builtin_ntp_servers=127.10.110.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:41:58.726231 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:58.713662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:45317 (local address 127.10.110.62:44923)
0926 16:41:58.714141 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:58.714146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:58.715345 (+ 1199us) server_negotiation.cc:407] Connection header received
0926 16:41:58.717646 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:58.717650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:58.717727 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:58.717814 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:58.719359 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:58.720250 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:58.722772 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:58.723261 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:58.723410 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:58.725342 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:58.725348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:58.725352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:58.725376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:58.725422 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:58.725437 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:58.725522 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:58.725821 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 16:41:58.725925 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":228,"threads_started":1}
I20260926 16:41:58.727664 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:58.714107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44923 (local address 127.10.110.1:45317)
0926 16:41:58.715144 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:58.715189 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:58.717067 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:58.718167 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:58.718181 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:58.718321 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:58.719112 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:58.719173 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:58.720431 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:58.720437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:58.722532 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:58.722553 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:58.723565 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:58.723573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:58.723961 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:58.724842 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:41:58.724846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:58.724910 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:41:58.725036 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:58.725543 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:58.725557 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:58.725565 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 16:41:58.725778 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 16:41:58.726269 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":296,"threads_started":1}
I20260926 16:41:58.730019 29630 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44923
I20260926 16:41:58.730387 29630 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:58.731244 29630 heartbeater.cc:511] Master 127.10.110.62:44923 requested a full tablet report, sending...
I20260926 16:41:58.733796 29444 ts_manager.cc:194] Registered new tserver with Master: e542e6d3cb6746e680bf5953c8dbc7ba (127.10.110.1:42115)
I20260926 16:41:58.735459 29444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.1:45317
W20260926 16:41:59.090361 29634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:59.090770 29634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:59.090914 29634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:59.102905 29634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:59.103029 29634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:59.103138 29634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:41:59.113018 29634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:59.114382 29634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:59.116330 29634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:59.131327 29639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:59.131386 29640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:59.131299 29642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:59.132426 29634 server_base.cc:1034] running on GCE node
I20260926 16:41:59.133184 29634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:59.134356 29634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:59.135638 29634 hybrid_clock.cc:648] HybridClock initialized: now 1790440919135544 us; error 88 us; skew 500 ppm
I20260926 16:41:59.136103 29634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:59.138816 29634 webserver.cc:460] Webserver started at http://127.10.110.2:35003/ using document root <none> and password file <none>
I20260926 16:41:59.139578 29634 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:59.139693 29634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:59.140084 29634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:59.143379 29634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "56430f5c536f4bdbb6e7ade73d2b10cc"
format_stamp: "Formatted at 2026-09-26 16:41:59 on dist-test-slave-whjp"
server_key: "441b6a98f59a92665809b8667e4e994f"
server_key_iv: "545f760da8a93b2d04e4153475651e8d"
server_key_version: "encryptionkey@0"
I20260926 16:41:59.144313 29634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "56430f5c536f4bdbb6e7ade73d2b10cc"
format_stamp: "Formatted at 2026-09-26 16:41:59 on dist-test-slave-whjp"
server_key: "441b6a98f59a92665809b8667e4e994f"
server_key_iv: "545f760da8a93b2d04e4153475651e8d"
server_key_version: "encryptionkey@0"
I20260926 16:41:59.150337 29634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:41:59.154564 29648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:59.156554 29634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:41:59.156740 29634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "56430f5c536f4bdbb6e7ade73d2b10cc"
format_stamp: "Formatted at 2026-09-26 16:41:59 on dist-test-slave-whjp"
server_key: "441b6a98f59a92665809b8667e4e994f"
server_key_iv: "545f760da8a93b2d04e4153475651e8d"
server_key_version: "encryptionkey@0"
I20260926 16:41:59.156987 29634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:59.175746 29634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:59.176677 29634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:59.176970 29634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:59.178035 29634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:59.180290 29634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:59.180400 29634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:59.180492 29634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:59.180546 29634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:59.216289 29634 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:33227
I20260926 16:41:59.216332 29760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:33227 every 8 connection(s)
I20260926 16:41:59.217685 29634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:41:59.219893 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29634
I20260926 16:41:59.220073 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:41:59.220618 10680 external_mini_cluster.cc:1428] Setting key 6e3140b2dfb0b84c7223924c5464b365
I20260926 16:41:59.223886 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44923
--builtin_ntp_servers=127.10.110.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:41:59.232522 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.220822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:39023 (local address 127.10.110.62:44923)
0926 16:41:59.221055 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.221060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.222647 (+ 1587us) server_negotiation.cc:407] Connection header received
0926 16:41:59.224732 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.224738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.224827 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.224934 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:59.226196 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.227308 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.229411 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.230036 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.230166 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.231654 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:59.231660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:59.231663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:59.231688 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:59.231722 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:59.231734 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:59.231805 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.232160 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.232262 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:41:59.234187 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.221477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44923 (local address 127.10.110.2:39023)
0926 16:41:59.222474 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.222519 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.224420 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.225205 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.225216 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.225320 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:59.225960 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.225979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.227490 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.227495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.229223 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.229240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.230312 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.230319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.230554 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.231289 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:41:59.231292 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:59.231339 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:41:59.231434 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:59.231857 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:59.231870 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:59.231879 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.232100 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.232588 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":312,"threads_started":1}
I20260926 16:41:59.235958 29761 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44923
I20260926 16:41:59.236308 29761 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:59.237164 29761 heartbeater.cc:511] Master 127.10.110.62:44923 requested a full tablet report, sending...
I20260926 16:41:59.238806 29444 ts_manager.cc:194] Registered new tserver with Master: 56430f5c536f4bdbb6e7ade73d2b10cc (127.10.110.2:33227)
I20260926 16:41:59.239949 29444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.2:39023
W20260926 16:41:59.642674 29764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:41:59.643002 29764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:41:59.643111 29764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:41:59.651261 29764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:41:59.651399 29764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:41:59.651540 29764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:41:59.662082 29764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:41:59.663511 29764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:41:59.665910 29764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:41:59.681937 29769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:59.681957 29772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:41:59.682005 29770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:41:59.682783 29764 server_base.cc:1034] running on GCE node
I20260926 16:41:59.683444 29764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:41:59.684638 29764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:41:59.685917 29764 hybrid_clock.cc:648] HybridClock initialized: now 1790440919685849 us; error 54 us; skew 500 ppm
I20260926 16:41:59.686333 29764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:41:59.689317 29764 webserver.cc:460] Webserver started at http://127.10.110.3:45507/ using document root <none> and password file <none>
I20260926 16:41:59.690100 29764 fs_manager.cc:359] Metadata directory not provided
I20260926 16:41:59.690220 29764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:41:59.690563 29764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:41:59.693787 29764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "31acef3eac7f4593aaa9525b3f09367a"
format_stamp: "Formatted at 2026-09-26 16:41:59 on dist-test-slave-whjp"
server_key: "22fee37246d222782f2a6d64aae35c92"
server_key_iv: "bc5dfe8ee09d57c7fc00b814059bd75e"
server_key_version: "encryptionkey@0"
I20260926 16:41:59.694751 29764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "31acef3eac7f4593aaa9525b3f09367a"
format_stamp: "Formatted at 2026-09-26 16:41:59 on dist-test-slave-whjp"
server_key: "22fee37246d222782f2a6d64aae35c92"
server_key_iv: "bc5dfe8ee09d57c7fc00b814059bd75e"
server_key_version: "encryptionkey@0"
I20260926 16:41:59.700968 29764 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 16:41:59.705140 29778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:59.706805 29764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:41:59.707005 29764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "31acef3eac7f4593aaa9525b3f09367a"
format_stamp: "Formatted at 2026-09-26 16:41:59 on dist-test-slave-whjp"
server_key: "22fee37246d222782f2a6d64aae35c92"
server_key_iv: "bc5dfe8ee09d57c7fc00b814059bd75e"
server_key_version: "encryptionkey@0"
I20260926 16:41:59.707257 29764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:41:59.723373 29764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:41:59.724408 29764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:41:59.724735 29764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:41:59.725883 29764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:41:59.727561 29764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:41:59.727670 29764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:41:59.727801 29764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:41:59.727902 29764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 16:41:59.739632 29630 heartbeater.cc:503] Master 127.10.110.62:44923 was elected leader, sending a full tablet report...
I20260926 16:41:59.760536 29764 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:33089
I20260926 16:41:59.760618 29890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:33089 every 8 connection(s)
I20260926 16:41:59.761943 29764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:41:59.766003 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29764
I20260926 16:41:59.766181 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:41:59.766616 10680 external_mini_cluster.cc:1428] Setting key 08d4c9586cf808520500474e80c976b8
I20260926 16:41:59.775758 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.764819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:45855 (local address 127.10.110.62:44923)
0926 16:41:59.765325 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.765330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.766878 (+ 1548us) server_negotiation.cc:407] Connection header received
0926 16:41:59.768778 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.768785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.768874 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.768967 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:59.770136 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.770905 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.772652 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.773068 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.773194 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.774770 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:59.774775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:59.774778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:59.774800 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:59.774843 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:59.774855 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:59.774944 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.775222 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.775342 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":293,"threads_started":1}
I20260926 16:41:59.776890 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.765508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:44923 (local address 127.10.110.3:45855)
0926 16:41:59.766636 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.766683 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.768493 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.769238 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.769247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.769353 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:41:59.769956 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.769975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.771073 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.771077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.772474 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.772491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.773327 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.773335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.773598 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.774372 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:41:59.774375 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:41:59.774421 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:41:59.774516 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:41:59.774960 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:41:59.774971 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:41:59.774980 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.775192 (+ 212us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.775666 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":315,"threads_started":1}
I20260926 16:41:59.778632 29891 heartbeater.cc:347] Connected to a master server at 127.10.110.62:44923
I20260926 16:41:59.778961 29891 heartbeater.cc:464] Registering TS with master...
I20260926 16:41:59.779872 29891 heartbeater.cc:511] Master 127.10.110.62:44923 requested a full tablet report, sending...
I20260926 16:41:59.781828 29444 ts_manager.cc:194] Registered new tserver with Master: 31acef3eac7f4593aaa9525b3f09367a (127.10.110.3:33089)
I20260926 16:41:59.782737 29444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.3:45855
I20260926 16:41:59.795217 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:41:59.806313 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.799611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.10.110.62:44923)
0926 16:41:59.799797 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.799801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.800187 (+ 386us) server_negotiation.cc:407] Connection header received
0926 16:41:59.800384 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.800388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.800475 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.800618 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:41:59.801178 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.801977 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.803620 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.804089 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.804215 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.805583 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:41:59.805588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:41:59.805591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:41:59.805614 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:41:59.805652 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:41:59.805665 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:41:59.805795 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.805993 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.806046 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:41:59.811247 29444 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:41:59.819906 29444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:41:59.848467 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.834381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:33089 (local address 127.0.0.1:54164)
0926 16:41:59.836758 (+ 2377us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.836793 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.836992 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.838021 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.838029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.838115 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:59.838828 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.838847 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.841233 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.841240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.844702 (+ 3462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.844723 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.847041 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.847049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.847327 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.847515 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.847725 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.848072 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2136,"mutex_wait_us":390,"thread_start_us":1994,"threads_started":1}
I20260926 16:41:59.848663 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.834510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.10.110.3:33089)
0926 16:41:59.835592 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.835603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.836920 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 16:41:59.837221 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.837231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.837518 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.837711 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:59.839112 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.840961 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.845007 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.846590 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.846739 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.847040 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.847859 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.848204 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":692,"threads_started":1}
I20260926 16:41:59.849144 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.835786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.10.110.1:42115)
0926 16:41:59.836411 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.836420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.837919 (+ 1499us) server_negotiation.cc:407] Connection header received
0926 16:41:59.838027 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.838035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.838428 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.838757 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:59.839754 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.841425 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.845647 (+ 4222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.847140 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.847311 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.847769 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.848618 (+ 849us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.848824 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260926 16:41:59.849637 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.834921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:42115 (local address 127.0.0.1:54354)
0926 16:41:59.837566 (+ 2645us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.837584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.837726 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.839018 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.839022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.839052 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:59.839443 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.839456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.841613 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.841619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.845085 (+ 3466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.845100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.847945 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.847953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.848193 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.848227 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.848316 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.848478 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2519,"mutex_wait_us":1015,"thread_start_us":2353,"threads_started":1}
I20260926 16:41:59.853878 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.838385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:33227 (local address 127.0.0.1:37582)
0926 16:41:59.843469 (+ 5084us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.843495 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.843659 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.844788 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.844793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.844820 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:59.845238 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.845252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.847189 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.847193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.850710 (+ 3517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.850734 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.852869 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.852876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.853065 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.853105 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.853215 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.853380 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4890,"mutex_wait_us":1,"thread_start_us":4669,"threads_started":1}
I20260926 16:41:59.853925 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.837901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.10.110.2:33227)
0926 16:41:59.844209 (+ 6308us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.844217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.844246 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:41:59.844332 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.844340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.844563 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.844768 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:59.845448 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.846994 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.851014 (+ 4020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.852608 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.852782 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.853061 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.853277 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.853534 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6141,"thread_start_us":6018,"threads_started":1}
I20260926 16:41:59.853889 29565 tablet_service.cc:1467] Processing CreateTablet for tablet 763b0bc0cd63440eac05ef53e55f6744 (DEFAULT_TABLE table=test-table [id=af4ad409ded946b29e3f1bb16418986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:59.853864 29826 tablet_service.cc:1467] Processing CreateTablet for tablet 763b0bc0cd63440eac05ef53e55f6744 (DEFAULT_TABLE table=test-table [id=af4ad409ded946b29e3f1bb16418986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:59.855667 29565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b0bc0cd63440eac05ef53e55f6744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:59.856179 29826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b0bc0cd63440eac05ef53e55f6744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:59.859207 29696 tablet_service.cc:1467] Processing CreateTablet for tablet 763b0bc0cd63440eac05ef53e55f6744 (DEFAULT_TABLE table=test-table [id=af4ad409ded946b29e3f1bb16418986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:41:59.861037 29696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763b0bc0cd63440eac05ef53e55f6744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:41:59.872414 29910 tablet_bootstrap.cc:492] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Bootstrap starting.
I20260926 16:41:59.873769 29911 tablet_bootstrap.cc:492] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Bootstrap starting.
I20260926 16:41:59.878049 29911 tablet_bootstrap.cc:654] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:59.878049 29910 tablet_bootstrap.cc:654] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:59.878892 29912 tablet_bootstrap.cc:492] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Bootstrap starting.
I20260926 16:41:59.879453 29910 log.cc:824] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:59.879451 29911 log.cc:824] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:59.883450 29912 tablet_bootstrap.cc:654] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Neither blocks nor log segments found. Creating new log.
I20260926 16:41:59.883741 29910 tablet_bootstrap.cc:492] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: No bootstrap required, opened a new log
I20260926 16:41:59.884047 29910 ts_tablet_manager.cc:1397] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260926 16:41:59.884855 29912 log.cc:824] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:41:59.888538 29912 tablet_bootstrap.cc:492] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: No bootstrap required, opened a new log
I20260926 16:41:59.888914 29912 ts_tablet_manager.cc:1397] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260926 16:41:59.894080 29910 raft_consensus.cc:348] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.894709 29910 raft_consensus.cc:374] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:59.894840 29910 raft_consensus.cc:729] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31acef3eac7f4593aaa9525b3f09367a, State: Initialized, Role: FOLLOWER
I20260926 16:41:59.894940 29911 tablet_bootstrap.cc:492] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: No bootstrap required, opened a new log
I20260926 16:41:59.895287 29911 ts_tablet_manager.cc:1397] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20260926 16:41:59.895570 29910 consensus_queue.cc:260] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.896944 29891 heartbeater.cc:503] Master 127.10.110.62:44923 was elected leader, sending a full tablet report...
I20260926 16:41:59.897868 29910 ts_tablet_manager.cc:1428] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 16:41:59.898427 29912 raft_consensus.cc:348] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.898972 29912 raft_consensus.cc:374] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:59.899237 29912 raft_consensus.cc:729] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56430f5c536f4bdbb6e7ade73d2b10cc, State: Initialized, Role: FOLLOWER
I20260926 16:41:59.900058 29912 consensus_queue.cc:260] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.901280 29761 heartbeater.cc:503] Master 127.10.110.62:44923 was elected leader, sending a full tablet report...
I20260926 16:41:59.902030 29912 ts_tablet_manager.cc:1428] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260926 16:41:59.905910 29911 raft_consensus.cc:348] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.906445 29911 raft_consensus.cc:374] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:41:59.906570 29911 raft_consensus.cc:729] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e542e6d3cb6746e680bf5953c8dbc7ba, State: Initialized, Role: FOLLOWER
I20260926 16:41:59.907325 29911 consensus_queue.cc:260] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.909343 29911 ts_tablet_manager.cc:1428] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260926 16:41:59.939263 29917 raft_consensus.cc:482] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:41:59.939568 29917 raft_consensus.cc:504] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.941610 29917 leader_election.cc:288] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31acef3eac7f4593aaa9525b3f09367a (127.10.110.3:33089), e542e6d3cb6746e680bf5953c8dbc7ba (127.10.110.1:42115)
I20260926 16:41:59.949783 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.942120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:49021 (local address 127.10.110.1:42115)
0926 16:41:59.942353 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.942361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.943361 (+ 1000us) server_negotiation.cc:407] Connection header received
0926 16:41:59.943626 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.943631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.943707 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.943785 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:59.944577 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.945281 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.947934 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.948842 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.949028 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.949114 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.949390 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.949526 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 16:41:59.949970 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.941954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:59033 (local address 127.10.110.3:33089)
0926 16:41:59.942365 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 16:41:59.942372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:41:59.943160 (+ 788us) server_negotiation.cc:407] Connection header received
0926 16:41:59.943462 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:41:59.943469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:41:59.943549 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:41:59.943614 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:41:59.944493 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.945369 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.947873 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.948946 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.949248 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.949320 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:41:59.949595 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 16:41:59.949728 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260926 16:41:59.949893 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.942254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:42115 (local address 127.10.110.2:49021)
0926 16:41:59.943246 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.943267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.943476 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.943815 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.943819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.943901 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:59.944401 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.944412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.945556 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.945563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.947687 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.947703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.949060 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.949067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.949223 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.949254 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.949342 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.949489 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":170,"spinlock_wait_cycles":34560,"thread_start_us":691,"threads_started":1}
I20260926 16:41:59.950455 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:41:59.942014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:33089 (local address 127.10.110.2:59033)
0926 16:41:59.943033 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0926 16:41:59.943071 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:41:59.943285 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:41:59.943805 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:41:59.943809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:41:59.943859 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:41:59.944333 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.944347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.945568 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.945573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.947506 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:41:59.947528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:41:59.949266 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:41:59.949273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:41:59.949440 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:41:59.949464 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:41:59.949559 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 16:41:59.949696 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":57,"thread_start_us":625,"threads_started":1}
I20260926 16:41:59.951438 29585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b0bc0cd63440eac05ef53e55f6744" candidate_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" is_pre_election: true
I20260926 16:41:59.951648 29846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b0bc0cd63440eac05ef53e55f6744" candidate_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31acef3eac7f4593aaa9525b3f09367a" is_pre_election: true
I20260926 16:41:59.952082 29585 raft_consensus.cc:2454] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56430f5c536f4bdbb6e7ade73d2b10cc in term 0.
I20260926 16:41:59.952318 29846 raft_consensus.cc:2454] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56430f5c536f4bdbb6e7ade73d2b10cc in term 0.
I20260926 16:41:59.952822 29651 leader_election.cc:302] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56430f5c536f4bdbb6e7ade73d2b10cc, e542e6d3cb6746e680bf5953c8dbc7ba; no voters:
I20260926 16:41:59.953317 29917 raft_consensus.cc:2781] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:41:59.953471 29917 raft_consensus.cc:482] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:41:59.953567 29917 raft_consensus.cc:3037] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:59.956198 29917 raft_consensus.cc:504] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
I20260926 16:41:59.956952 29917 leader_election.cc:288] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [CANDIDATE]: Term 1 election: Requested vote from peers 31acef3eac7f4593aaa9525b3f09367a (127.10.110.3:33089), e542e6d3cb6746e680bf5953c8dbc7ba (127.10.110.1:42115)
I20260926 16:41:59.957620 29846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b0bc0cd63440eac05ef53e55f6744" candidate_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31acef3eac7f4593aaa9525b3f09367a"
I20260926 16:41:59.957720 29585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763b0bc0cd63440eac05ef53e55f6744" candidate_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba"
I20260926 16:41:59.957844 29846 raft_consensus.cc:3037] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:59.957974 29585 raft_consensus.cc:3037] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:41:59.960696 29846 raft_consensus.cc:2454] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56430f5c536f4bdbb6e7ade73d2b10cc in term 1.
I20260926 16:41:59.960691 29585 raft_consensus.cc:2454] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56430f5c536f4bdbb6e7ade73d2b10cc in term 1.
I20260926 16:41:59.961355 29649 leader_election.cc:302] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31acef3eac7f4593aaa9525b3f09367a, 56430f5c536f4bdbb6e7ade73d2b10cc, e542e6d3cb6746e680bf5953c8dbc7ba; no voters:
I20260926 16:41:59.961860 29917 raft_consensus.cc:2781] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:41:59.962355 29917 raft_consensus.cc:686] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 1 LEADER]: Becoming Leader. State: Replica: 56430f5c536f4bdbb6e7ade73d2b10cc, State: Running, Role: LEADER
I20260926 16:41:59.962986 29917 consensus_queue.cc:237] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } }
W20260926 16:41:59.963651 29631 tablet.cc:2369] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:41:59.969107 29443 catalog_manager.cc:5495] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 56430f5c536f4bdbb6e7ade73d2b10cc (127.10.110.2). New cstate: current_term: 1 leader_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56430f5c536f4bdbb6e7ade73d2b10cc" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 33227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 } health_report { overall_health: UNKNOWN } } }
W20260926 16:41:59.969846 29762 tablet.cc:2369] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:00.013988 29892 tablet.cc:2369] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:00.015516 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:00.008779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.10.110.2:33227)
0926 16:42:00.009000 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:00.009007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:00.009070 (+ 63us) server_negotiation.cc:407] Connection header received
0926 16:42:00.009336 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:00.009340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:00.009444 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:00.009585 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:00.010033 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.010770 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.012537 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.013043 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.013182 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:00.013933 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:00.014662 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:00.014876 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:00.015178 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:42:00.015223 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:42:00.025818 29585 raft_consensus.cc:1261] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 1 FOLLOWER]: Refusing update from remote peer 56430f5c536f4bdbb6e7ade73d2b10cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:00.025799 29846 raft_consensus.cc:1261] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 1 FOLLOWER]: Refusing update from remote peer 56430f5c536f4bdbb6e7ade73d2b10cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:00.027271 29917 consensus_queue.cc:1035] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e542e6d3cb6746e680bf5953c8dbc7ba" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 42115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:00.027740 29921 consensus_queue.cc:1035] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "31acef3eac7f4593aaa9525b3f09367a" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:00.042963 29924 mvcc.cc:205] Tried to move back new op lower bound from 7333646008411299840 to 7333646008174198784. Current Snapshot: MvccSnapshot[applied={T|T < 7333646008411299840}]
I20260926 16:42:00.044967 29925 mvcc.cc:205] Tried to move back new op lower bound from 7333646008411299840 to 7333646008174198784. Current Snapshot: MvccSnapshot[applied={T|T < 7333646008411299840}]
I20260926 16:42:00.059648 29443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:00.059990 29443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:00.064844 29443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4095c09c783e4116aa05b5a5e413c052: Sending DeleteTablet for 3 replicas of tablet 763b0bc0cd63440eac05ef53e55f6744
I20260926 16:42:00.067198 29826 tablet_service.cc:1514] Processing DeleteTablet for tablet 763b0bc0cd63440eac05ef53e55f6744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:00 UTC) from {username='slave'} at 127.0.0.1:54164
I20260926 16:42:00.067122 29565 tablet_service.cc:1514] Processing DeleteTablet for tablet 763b0bc0cd63440eac05ef53e55f6744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:00 UTC) from {username='slave'} at 127.0.0.1:54354
I20260926 16:42:00.067595 29696 tablet_service.cc:1514] Processing DeleteTablet for tablet 763b0bc0cd63440eac05ef53e55f6744 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:00 UTC) from {username='slave'} at 127.0.0.1:37582
I20260926 16:42:00.068846 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29503
I20260926 16:42:00.068845 29934 tablet_replica.cc:330] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: stopping tablet replica
I20260926 16:42:00.069459 29934 raft_consensus.cc:2229] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:00.070971 29934 raft_consensus.cc:2258] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:00.072106 29933 tablet_replica.cc:330] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: stopping tablet replica
I20260926 16:42:00.072636 29933 raft_consensus.cc:2229] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:00.073344 29933 raft_consensus.cc:2258] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:00.075510 29934 ts_tablet_manager.cc:1905] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:00.076799 29933 ts_tablet_manager.cc:1905] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:00.083101 29934 ts_tablet_manager.cc:1918] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:00.083420 29934 log.cc:1196] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/763b0bc0cd63440eac05ef53e55f6744
I20260926 16:42:00.084040 29935 tablet_replica.cc:330] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: stopping tablet replica
I20260926 16:42:00.084218 29934 ts_tablet_manager.cc:1939] T 763b0bc0cd63440eac05ef53e55f6744 P e542e6d3cb6746e680bf5953c8dbc7ba: Deleting consensus metadata
I20260926 16:42:00.084569 29935 raft_consensus.cc:2229] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:00.084873 29933 ts_tablet_manager.cc:1918] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:00.085071 29933 log.cc:1196] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/763b0bc0cd63440eac05ef53e55f6744
I20260926 16:42:00.085345 29935 raft_consensus.cc:2258] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:00.085590 29933 ts_tablet_manager.cc:1939] T 763b0bc0cd63440eac05ef53e55f6744 P 31acef3eac7f4593aaa9525b3f09367a: Deleting consensus metadata
I20260926 16:42:00.087512 29430 catalog_manager.cc:4841] TS e542e6d3cb6746e680bf5953c8dbc7ba (127.10.110.1:42115): tablet 763b0bc0cd63440eac05ef53e55f6744 (table test-table [id=af4ad409ded946b29e3f1bb16418986e]) successfully deleted
I20260926 16:42:00.087875 29935 ts_tablet_manager.cc:1905] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:00.094539 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:00.072786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.10.110.1:42115)
0926 16:42:00.072982 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:00.072988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:00.074330 (+ 1342us) server_negotiation.cc:407] Connection header received
0926 16:42:00.074524 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:00.074529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:00.074645 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:00.074751 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:00.086082 (+ 11331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.087182 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.089925 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.090609 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.090725 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:00.092466 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:00.092479 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:00.092500 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:00.093363 (+ 863us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:00.093438 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:00.093470 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:00.093572 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:00.093834 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 16:42:00.093976 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:42:00.095789 29625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:00.096335 29935 ts_tablet_manager.cc:1918] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:00.096556 29935 log.cc:1196] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/763b0bc0cd63440eac05ef53e55f6744
I20260926 16:42:00.097182 29935 ts_tablet_manager.cc:1939] T 763b0bc0cd63440eac05ef53e55f6744 P 56430f5c536f4bdbb6e7ade73d2b10cc: Deleting consensus metadata
I20260926 16:42:00.100055 29428 catalog_manager.cc:4841] TS 56430f5c536f4bdbb6e7ade73d2b10cc (127.10.110.2:33227): tablet 763b0bc0cd63440eac05ef53e55f6744 (table test-table [id=af4ad409ded946b29e3f1bb16418986e]) successfully deleted
I20260926 16:42:00.103340 29428 catalog_manager.cc:4841] TS 31acef3eac7f4593aaa9525b3f09367a (127.10.110.3:33089): tablet 763b0bc0cd63440eac05ef53e55f6744 (table test-table [id=af4ad409ded946b29e3f1bb16418986e]) successfully deleted
I20260926 16:42:00.191603 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29503
I20260926 16:42:00.207309 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29634
I20260926 16:42:00.215942 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:00.208070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.10.110.2:33227)
0926 16:42:00.208361 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:00.208367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:00.208391 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:42:00.208495 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:00.208505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:00.208621 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:00.208754 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:00.209091 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.210338 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.211952 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.212721 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.212860 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:00.213997 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:00.214009 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:00.214030 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:00.214887 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:00.214958 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:00.214993 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:00.215119 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:00.215317 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:42:00.215443 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260926 16:42:00.216889 29756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:00.334502 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29634
I20260926 16:42:00.351092 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29764
I20260926 16:42:00.358134 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:00.351944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.10.110.3:33089)
0926 16:42:00.352127 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:00.352133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:00.352155 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:42:00.352339 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:00.352344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:00.352436 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:00.352518 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:00.352936 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.353694 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.355097 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:00.355500 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:00.355612 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:00.356760 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:00.356769 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:00.356784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:00.357375 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:00.357428 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:00.357450 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:00.357559 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:00.357705 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:42:00.357827 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:42:00.358912 29886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:00.457055 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29764
I20260926 16:42:00.470816 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29413
I20260926 16:42:00.472075 29474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:00.586068 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29413
2026-09-26T16:42:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3059 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:00 dist-test-slave-whjp krb5kdc[29943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:00 dist-test-slave-whjp krb5kdc[29943](info): set up 2 sockets
Sep 26 16:42:00 dist-test-slave-whjp krb5kdc[29943](info): commencing operation
krb5kdc: starting...
W20260926 16:42:01.688925 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:01 dist-test-slave-whjp krb5kdc[29943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:01.981814 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:43343
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:02.433138 29956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:02.433498 29956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:02.433636 29956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:02.442147 29956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:02.442332 29956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:02.442412 29956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:02.442490 29956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:02.442562 29956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:02.453269 29956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:43343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:02.454728 29956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:02.456156 29956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:02.469975 29964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:02.469942 29962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:02.470150 29961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:02.470830 29956 server_base.cc:1034] running on GCE node
I20260926 16:42:02.471699 29956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:02.473431 29956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:02.474694 29956 hybrid_clock.cc:648] HybridClock initialized: now 1790440922474634 us; error 53 us; skew 500 ppm
I20260926 16:42:02.475134 29956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:02 dist-test-slave-whjp krb5kdc[29943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440922, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:02.478904 29956 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:02.480782 29956 webserver.cc:460] Webserver started at http://127.10.110.62:41349/ using document root <none> and password file <none>
I20260926 16:42:02.481571 29956 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:02.481707 29956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:02.481992 29956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:02.485603 29956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "b3245dd33c6248d5b1ce9b34c10cdd91"
format_stamp: "Formatted at 2026-09-26 16:42:02 on dist-test-slave-whjp"
server_key: "fd0bf33740a871367a0c3e67cc5a347b"
server_key_iv: "fd5f96c8722a83f4e0eacd547bff1e72"
server_key_version: "encryptionkey@0"
I20260926 16:42:02.486485 29956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "b3245dd33c6248d5b1ce9b34c10cdd91"
format_stamp: "Formatted at 2026-09-26 16:42:02 on dist-test-slave-whjp"
server_key: "fd0bf33740a871367a0c3e67cc5a347b"
server_key_iv: "fd5f96c8722a83f4e0eacd547bff1e72"
server_key_version: "encryptionkey@0"
I20260926 16:42:02.493674 29956 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 16:42:02.498234 29971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:02.500133 29956 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:42:02.500375 29956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "b3245dd33c6248d5b1ce9b34c10cdd91"
format_stamp: "Formatted at 2026-09-26 16:42:02 on dist-test-slave-whjp"
server_key: "fd0bf33740a871367a0c3e67cc5a347b"
server_key_iv: "fd5f96c8722a83f4e0eacd547bff1e72"
server_key_version: "encryptionkey@0"
I20260926 16:42:02.500615 29956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:02.521790 29956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:02.527379 29956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:02.527710 29956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:02.547611 29956 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:43343
I20260926 16:42:02.547613 30022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:43343 every 8 connection(s)
I20260926 16:42:02.549165 29956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:02.554113 30023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:02.555044 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 29956
I20260926 16:42:02.555227 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:02.555723 10680 external_mini_cluster.cc:1428] Setting key d721d91d6a825b1c5026144de6701e51
I20260926 16:42:02.570065 30023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91: Bootstrap starting.
Sep 26 16:42:02 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:02.574311 30023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:02.575870 30023 log.cc:824] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:02.580062 30023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91: No bootstrap required, opened a new log
I20260926 16:42:02.585433 30026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:02.557793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.10.110.62:43343)
0926 16:42:02.558709 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:02.558728 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:02.558788 (+ 60us) server_negotiation.cc:407] Connection header received
0926 16:42:02.561305 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:02.561349 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:02.562079 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:02.562633 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:02.564252 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:02.566320 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:02.568052 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:02.569223 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:02.569448 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:02.575076 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:02.575089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:02.575113 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:02.575183 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:02.579470 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:02.580659 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:02.580669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:02.580680 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:02.580942 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:02.581586 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:02.581591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:02.581595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:02.582428 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:02.582770 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:02.583217 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 16:42:02.583777 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":355,"threads_started":1}
I20260926 16:42:02.590580 30023 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 43343 } }
I20260926 16:42:02.591028 30023 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:02.591109 30023 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3245dd33c6248d5b1ce9b34c10cdd91, State: Initialized, Role: FOLLOWER
I20260926 16:42:02.591725 30023 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 43343 } }
I20260926 16:42:02.592017 30023 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:02.592181 30023 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:02.592360 30023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:02.594810 30023 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 43343 } }
I20260926 16:42:02.595440 30023 leader_election.cc:302] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3245dd33c6248d5b1ce9b34c10cdd91; no voters:
I20260926 16:42:02.596268 30023 leader_election.cc:288] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:02.596465 30028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:02.597218 30028 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [term 1 LEADER]: Becoming Leader. State: Replica: b3245dd33c6248d5b1ce9b34c10cdd91, State: Running, Role: LEADER
I20260926 16:42:02.597872 30028 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 43343 } }
I20260926 16:42:02.598793 30023 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:02.601120 30030 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3245dd33c6248d5b1ce9b34c10cdd91. Latest consensus state: current_term: 1 leader_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 43343 } } }
I20260926 16:42:02.601105 30029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3245dd33c6248d5b1ce9b34c10cdd91" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 43343 } } }
I20260926 16:42:02.601663 30030 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:02.601845 30029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:02.604102 30035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:02.611058 30035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:02.622186 30035 catalog_manager.cc:1347] Generated new cluster ID: 6c54c016c55f4860b26eac4e7dd1b829
I20260926 16:42:02.622339 30035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:02.652397 30035 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:02.654464 30035 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:02.663371 30035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91: Generated new TSK 0
I20260926 16:42:02.664322 30035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.110.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
Entry for principal kudu/127.10.110.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab.
I20260926 16:42:02.744323 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:43343
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:03.113497 30049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:03.113797 30049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:03.113891 30049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:03.121534 30049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:03.121670 30049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:03.121793 30049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:42:03.131994 30049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.1.keytab
--principal=kudu/127.10.110.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:03.133301 30049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:03.134656 30049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:03.149032 30054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:03.149003 30055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:03.148975 30057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:03.149992 30049 server_base.cc:1034] running on GCE node
I20260926 16:42:03.150746 30049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:03.151813 30049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:03.153040 30049 hybrid_clock.cc:648] HybridClock initialized: now 1790440923152977 us; error 62 us; skew 500 ppm
I20260926 16:42:03.153388 30049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:03 dist-test-slave-whjp krb5kdc[29943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440923, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:03.156747 30049 init.cc:377] Logged in from keytab as kudu/127.10.110.1@KRBTEST.COM (short username kudu)
I20260926 16:42:03.158419 30049 webserver.cc:460] Webserver started at http://127.10.110.1:40661/ using document root <none> and password file <none>
I20260926 16:42:03.159175 30049 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:03.159300 30049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:03.159622 30049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:03.162848 30049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "8b6866c56bdb47c1ad6e53b73d46339f"
format_stamp: "Formatted at 2026-09-26 16:42:03 on dist-test-slave-whjp"
server_key: "8c43f6c24f61d4f82d4165eebce3af51"
server_key_iv: "66d0c859eb3b36c5750b666c1db983f2"
server_key_version: "encryptionkey@0"
I20260926 16:42:03.163658 30049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "8b6866c56bdb47c1ad6e53b73d46339f"
format_stamp: "Formatted at 2026-09-26 16:42:03 on dist-test-slave-whjp"
server_key: "8c43f6c24f61d4f82d4165eebce3af51"
server_key_iv: "66d0c859eb3b36c5750b666c1db983f2"
server_key_version: "encryptionkey@0"
I20260926 16:42:03.169363 30049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:42:03.173164 30064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:03.174646 30049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:03.174810 30049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "8b6866c56bdb47c1ad6e53b73d46339f"
format_stamp: "Formatted at 2026-09-26 16:42:03 on dist-test-slave-whjp"
server_key: "8c43f6c24f61d4f82d4165eebce3af51"
server_key_iv: "66d0c859eb3b36c5750b666c1db983f2"
server_key_version: "encryptionkey@0"
I20260926 16:42:03.175012 30049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:03.194746 30049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:03.199815 30049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:03.200141 30049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:03.201193 30049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:03.202623 30049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:03.202708 30049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:03.202826 30049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:03.202875 30049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:03.233887 30049 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:41981
I20260926 16:42:03.233968 30176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:41981 every 8 connection(s)
I20260926 16:42:03.235155 30049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:03.245486 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30049
I20260926 16:42:03.245749 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:03.246256 10680 external_mini_cluster.cc:1428] Setting key a669dce8654bfed2076b4fc496c9857b
Sep 26 16:42:03 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440923, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.1@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:03.256482 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:03.237712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:56833 (local address 127.10.110.62:43343)
0926 16:42:03.238138 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:03.238144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:03.239136 (+ 992us) server_negotiation.cc:407] Connection header received
0926 16:42:03.241045 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:03.241051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:03.241140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:03.241239 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:03.242830 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.243562 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.245215 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.245621 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.245767 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:03.249547 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:03.249552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:03.249556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:03.249592 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:03.252423 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:03.253321 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:03.253330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:03.253332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:03.253468 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:03.253950 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:03.253953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:03.253956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:03.254416 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:03.254601 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:03.255705 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0926 16:42:03.256021 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20260926 16:42:03.257539 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:03.238258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:43343 (local address 127.10.110.1:56833)
0926 16:42:03.238982 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:03.239022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:03.240754 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:03.241501 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:03.241511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:03.242045 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:03.242642 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:03.242661 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.243723 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.243726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:03.245055 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:03.245070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.245890 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.245898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:03.246140 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:03.246876 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:03.246903 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:03.249296 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:03.252608 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:03.252616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:03.252632 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:03.253150 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:03.253590 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:03.253594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:03.253596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:03.253820 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:03.254601 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:03.254650 (+ 49us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:03.255152 (+ 502us) client_negotiation.cc:769] Sending connection context
0926 16:42:03.255514 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 16:42:03.256240 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":257,"threads_started":1}
I20260926 16:42:03.259351 30177 heartbeater.cc:347] Connected to a master server at 127.10.110.62:43343
I20260926 16:42:03.259701 30177 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:03.260583 30177 heartbeater.cc:511] Master 127.10.110.62:43343 requested a full tablet report, sending...
I20260926 16:42:03.263046 29988 ts_manager.cc:194] Registered new tserver with Master: 8b6866c56bdb47c1ad6e53b73d46339f (127.10.110.1:41981)
I20260926 16:42:03.264616 29988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.1@KRBTEST.COM'} at 127.10.110.1:56833
WARNING: no policy specified for kudu/127.10.110.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
Entry for principal kudu/127.10.110.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab.
I20260926 16:42:03.303563 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:43343
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:03.661330 30183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:03.661659 30183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:03.661760 30183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:03.670099 30183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:03.670279 30183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:03.670404 30183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:42:03.681681 30183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.2.keytab
--principal=kudu/127.10.110.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:03.683071 30183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:03.684688 30183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:03.700786 30188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:03.700685 30191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:03.700721 30189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:03.701226 30183 server_base.cc:1034] running on GCE node
I20260926 16:42:03.702340 30183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:03.703735 30183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:03.705097 30183 hybrid_clock.cc:648] HybridClock initialized: now 1790440923705059 us; error 41 us; skew 500 ppm
I20260926 16:42:03.705483 30183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:03 dist-test-slave-whjp krb5kdc[29943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440923, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:03.709097 30183 init.cc:377] Logged in from keytab as kudu/127.10.110.2@KRBTEST.COM (short username kudu)
I20260926 16:42:03.710829 30183 webserver.cc:460] Webserver started at http://127.10.110.2:46839/ using document root <none> and password file <none>
I20260926 16:42:03.711602 30183 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:03.711735 30183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:03.712093 30183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:03.715417 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "eb4bc497ccd743a5b0611ec5551fa00b"
format_stamp: "Formatted at 2026-09-26 16:42:03 on dist-test-slave-whjp"
server_key: "2c0d9163a0e22de376d1c7fa50320c2a"
server_key_iv: "7ac09d6549fb811b13af8b23b72f4a59"
server_key_version: "encryptionkey@0"
I20260926 16:42:03.716651 30183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "eb4bc497ccd743a5b0611ec5551fa00b"
format_stamp: "Formatted at 2026-09-26 16:42:03 on dist-test-slave-whjp"
server_key: "2c0d9163a0e22de376d1c7fa50320c2a"
server_key_iv: "7ac09d6549fb811b13af8b23b72f4a59"
server_key_version: "encryptionkey@0"
I20260926 16:42:03.722764 30183 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:03.727118 30198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:03.729111 30183 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:42:03.729329 30183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "eb4bc497ccd743a5b0611ec5551fa00b"
format_stamp: "Formatted at 2026-09-26 16:42:03 on dist-test-slave-whjp"
server_key: "2c0d9163a0e22de376d1c7fa50320c2a"
server_key_iv: "7ac09d6549fb811b13af8b23b72f4a59"
server_key_version: "encryptionkey@0"
I20260926 16:42:03.729552 30183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:03.766417 30183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:03.771654 30183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:03.771996 30183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:03.773092 30183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:03.774657 30183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:03.774745 30183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:03.774880 30183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:03.774935 30183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:03.807928 30183 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:45533
I20260926 16:42:03.807983 30310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:45533 every 8 connection(s)
I20260926 16:42:03.809484 30183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:03.819372 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30183
I20260926 16:42:03.819546 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:03.820117 10680 external_mini_cluster.cc:1428] Setting key 0627bb498ac807c95cfbedd07a182600
Sep 26 16:42:03 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440923, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.2@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:03.834769 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:03.812668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:42835 (local address 127.10.110.62:43343)
0926 16:42:03.813221 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:03.813226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:03.814314 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 16:42:03.816452 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:03.816459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:03.816627 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:03.816797 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:03.818679 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.819414 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.821820 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.822526 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.822665 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:03.827511 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:03.827519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:03.827523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:03.827572 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:03.830723 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:03.831766 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:03.831772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:03.831775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:03.831993 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:03.832540 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:03.832545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:03.832549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:03.832940 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:03.833153 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:03.834082 (+ 929us) server_negotiation.cc:299] Negotiation successful
0926 16:42:03.834351 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":239,"threads_started":1}
I20260926 16:42:03.835736 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:03.813311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:43343 (local address 127.10.110.2:42835)
0926 16:42:03.814108 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:03.814159 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:03.816105 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:03.817150 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:03.817163 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:03.817775 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:03.818441 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:03.818463 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.819641 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.819658 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:03.821634 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:03.821661 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:03.822826 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:03.822834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:03.823159 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:03.824086 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:03.824126 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:03.827145 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:03.830940 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:03.830950 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:03.830973 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:03.831599 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:03.832107 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:03.832111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:03.832114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:03.832384 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:03.833161 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:03.833176 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:03.833623 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 16:42:03.833924 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 16:42:03.834468 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":307,"threads_started":1}
I20260926 16:42:03.837446 30311 heartbeater.cc:347] Connected to a master server at 127.10.110.62:43343
I20260926 16:42:03.837817 30311 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:03.838711 30311 heartbeater.cc:511] Master 127.10.110.62:43343 requested a full tablet report, sending...
I20260926 16:42:03.840752 29988 ts_manager.cc:194] Registered new tserver with Master: eb4bc497ccd743a5b0611ec5551fa00b (127.10.110.2:45533)
I20260926 16:42:03.841688 29988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.2@KRBTEST.COM'} at 127.10.110.2:42835
WARNING: no policy specified for kudu/127.10.110.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
Entry for principal kudu/127.10.110.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab.
I20260926 16:42:03.883944 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:43343
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:04.225109 30317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:04.225447 30317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:04.225590 30317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:04.234647 30317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:04.234790 30317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:04.234963 30317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:42:04.245353 30317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.3.keytab
--principal=kudu/127.10.110.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:04.246672 30317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:04.248036 30317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:04.260722 30322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:04.260776 30325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:04.260739 30323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:04.261605 30317 server_base.cc:1034] running on GCE node
I20260926 16:42:04.262209 30317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:04.263386 30317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:04.264601 30317 hybrid_clock.cc:648] HybridClock initialized: now 1790440924264557 us; error 36 us; skew 500 ppm
I20260926 16:42:04.264952 30317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:04 dist-test-slave-whjp krb5kdc[29943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440924, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:04.268059 30317 init.cc:377] Logged in from keytab as kudu/127.10.110.3@KRBTEST.COM (short username kudu)
I20260926 16:42:04.268545 30177 heartbeater.cc:503] Master 127.10.110.62:43343 was elected leader, sending a full tablet report...
I20260926 16:42:04.269593 30317 webserver.cc:460] Webserver started at http://127.10.110.3:35273/ using document root <none> and password file <none>
I20260926 16:42:04.270298 30317 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:04.270414 30317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:04.270715 30317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:04.273679 30317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "6e0cd491554d4298af2a40d8ac40288f"
format_stamp: "Formatted at 2026-09-26 16:42:04 on dist-test-slave-whjp"
server_key: "482d3cec07a1ea3f2c128cf5705cf86f"
server_key_iv: "de0e55060e852b0737ebac4890b5a971"
server_key_version: "encryptionkey@0"
I20260926 16:42:04.274434 30317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "6e0cd491554d4298af2a40d8ac40288f"
format_stamp: "Formatted at 2026-09-26 16:42:04 on dist-test-slave-whjp"
server_key: "482d3cec07a1ea3f2c128cf5705cf86f"
server_key_iv: "de0e55060e852b0737ebac4890b5a971"
server_key_version: "encryptionkey@0"
I20260926 16:42:04.279985 30317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:42:04.283617 30332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:04.284999 30317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:04.285161 30317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "6e0cd491554d4298af2a40d8ac40288f"
format_stamp: "Formatted at 2026-09-26 16:42:04 on dist-test-slave-whjp"
server_key: "482d3cec07a1ea3f2c128cf5705cf86f"
server_key_iv: "de0e55060e852b0737ebac4890b5a971"
server_key_version: "encryptionkey@0"
I20260926 16:42:04.285353 30317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:04.296264 30317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:04.300374 30317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:04.300637 30317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:04.301573 30317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:04.302855 30317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:04.302929 30317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:04.303045 30317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:04.303097 30317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:04.333890 30317 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:42271
I20260926 16:42:04.333923 30444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:42271 every 8 connection(s)
I20260926 16:42:04.335225 30317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:04.339648 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30317
I20260926 16:42:04.339798 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:04.340250 10680 external_mini_cluster.cc:1428] Setting key 620716c62d8bc0150638a6df5a76d245
Sep 26 16:42:04 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440924, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.3@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:04.355650 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.337630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:42815 (local address 127.10.110.62:43343)
0926 16:42:04.338053 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.338058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.338919 (+ 861us) server_negotiation.cc:407] Connection header received
0926 16:42:04.340822 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.340828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.340913 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.341005 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:04.342666 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.343484 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.345072 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.345467 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.345573 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.349363 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:04.349368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:04.349371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:04.349408 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:04.352121 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.352902 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.352906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.352909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.353051 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.353578 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.353582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.353592 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.353895 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:04.354025 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.355066 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.355304 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":197,"threads_started":1}
I20260926 16:42:04.356671 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.338065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:43343 (local address 127.10.110.3:42815)
0926 16:42:04.338775 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:04.338810 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:04.340554 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:04.341277 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:04.341289 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:04.341876 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:04.342489 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.342507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.343626 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.343630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.344934 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.344946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.345678 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.345685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.345958 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.346645 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:04.346688 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:04.349122 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:04.352283 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:04.352294 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:04.352311 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:04.352765 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:04.353185 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:04.353189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:04.353192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:04.353417 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:04.354125 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:04.354151 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:04.354588 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 16:42:04.354891 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 16:42:04.355444 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260926 16:42:04.358036 30445 heartbeater.cc:347] Connected to a master server at 127.10.110.62:43343
I20260926 16:42:04.358366 30445 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:04.359071 30445 heartbeater.cc:511] Master 127.10.110.62:43343 requested a full tablet report, sending...
I20260926 16:42:04.360603 29988 ts_manager.cc:194] Registered new tserver with Master: 6e0cd491554d4298af2a40d8ac40288f (127.10.110.3:42271)
I20260926 16:42:04.361438 29988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.110.3@KRBTEST.COM'} at 127.10.110.3:42815
I20260926 16:42:04.368232 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:04.387120 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.373946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.10.110.62:43343)
0926 16:42:04.374130 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.374137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.374535 (+ 398us) server_negotiation.cc:407] Connection header received
0926 16:42:04.374760 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.374766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.374854 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.374955 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:04.376894 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.377884 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.379329 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.379805 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.379983 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.382198 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:04.382203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:04.382207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:04.382243 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:04.384746 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.385391 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.385395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.385397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.385574 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.385878 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.385881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.385883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.386215 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:04.386403 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.386703 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.386873 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:04.391186 29988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:04.397879 29988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:04.433475 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.409842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:42271 (local address 127.0.0.1:39830)
0926 16:42:04.419453 (+ 9611us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:04.419493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:04.419683 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:04.425900 (+ 6217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:04.425910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:04.425946 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:04.426481 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.426496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.428447 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.428453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.430696 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.430709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.432450 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.432456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.432641 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.432695 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 16:42:04.432831 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 16:42:04.433029 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9348,"mutex_wait_us":1,"thread_start_us":16992,"threads_started":1}
I20260926 16:42:04.433568 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.410160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.10.110.3:42271)
0926 16:42:04.424876 (+ 14716us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.424887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.424926 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:42:04.425037 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.425047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.425349 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.425563 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:04.426715 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.428314 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.430866 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.431924 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.432049 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.432323 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.433048 (+ 725us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.433229 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14497,"thread_start_us":14524,"threads_started":1}
I20260926 16:42:04.434712 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.411793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41981 (local address 127.0.0.1:37752)
0926 16:42:04.427072 (+ 15279us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:04.427103 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:04.427314 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:04.428135 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:04.428139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:04.428166 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:04.428617 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.428626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.429930 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.429935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.432142 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.432157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.434055 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.434065 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.434244 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.434277 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:42:04.434348 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 16:42:04.434438 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15167,"mutex_wait_us":21,"thread_start_us":7241,"threads_started":1}
I20260926 16:42:04.434882 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.412498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.10.110.1:41981)
0926 16:42:04.425388 (+ 12890us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.425398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.427211 (+ 1813us) server_negotiation.cc:407] Connection header received
0926 16:42:04.427483 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.427492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.427748 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.427957 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:04.428765 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.429795 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.432341 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.433785 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.433922 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.434139 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.434420 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.434568 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12733,"thread_start_us":12598,"threads_started":1}
I20260926 16:42:04.436200 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.410010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:45533 (local address 127.0.0.1:49838)
0926 16:42:04.424480 (+ 14470us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:04.424501 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:04.424621 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:04.427911 (+ 3290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:04.427915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:04.427938 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:04.428383 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.428399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.430223 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.430240 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.432378 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.432391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.435391 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.435397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.435556 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.435587 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:42:04.435691 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 16:42:04.435820 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14336,"mutex_wait_us":4,"thread_start_us":14174,"threads_started":1}
I20260926 16:42:04.436208 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.410548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.10.110.2:45533)
0926 16:42:04.427431 (+ 16883us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.427439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.427461 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:42:04.427533 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.427539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.427722 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.427906 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:04.428588 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.430086 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.432551 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.435193 (+ 2642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.435330 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.435501 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.435728 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.435907 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16718,"thread_start_us":16592,"threads_started":1}
I20260926 16:42:04.438143 30112 tablet_service.cc:1467] Processing CreateTablet for tablet 2e618d4961c34442a712b391978c209a (DEFAULT_TABLE table=test-table [id=f09f7e445dc34fa5bae6d951502ffeb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:04.437935 30380 tablet_service.cc:1467] Processing CreateTablet for tablet 2e618d4961c34442a712b391978c209a (DEFAULT_TABLE table=test-table [id=f09f7e445dc34fa5bae6d951502ffeb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:04.439435 30380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e618d4961c34442a712b391978c209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:04.439438 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e618d4961c34442a712b391978c209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:04.439759 30246 tablet_service.cc:1467] Processing CreateTablet for tablet 2e618d4961c34442a712b391978c209a (DEFAULT_TABLE table=test-table [id=f09f7e445dc34fa5bae6d951502ffeb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:04.441076 30246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e618d4961c34442a712b391978c209a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:04.450325 30464 tablet_bootstrap.cc:492] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Bootstrap starting.
I20260926 16:42:04.453323 30465 tablet_bootstrap.cc:492] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Bootstrap starting.
I20260926 16:42:04.453732 30464 tablet_bootstrap.cc:654] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:04.455018 30464 log.cc:824] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:04.456543 30466 tablet_bootstrap.cc:492] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Bootstrap starting.
I20260926 16:42:04.457084 30465 tablet_bootstrap.cc:654] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:04.458046 30464 tablet_bootstrap.cc:492] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: No bootstrap required, opened a new log
I20260926 16:42:04.458290 30464 ts_tablet_manager.cc:1397] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260926 16:42:04.458380 30465 log.cc:824] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:04.460458 30466 tablet_bootstrap.cc:654] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:04.461766 30466 log.cc:824] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:04.464341 30465 tablet_bootstrap.cc:492] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: No bootstrap required, opened a new log
I20260926 16:42:04.464550 30466 tablet_bootstrap.cc:492] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: No bootstrap required, opened a new log
I20260926 16:42:04.464598 30465 ts_tablet_manager.cc:1397] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260926 16:42:04.464812 30466 ts_tablet_manager.cc:1397] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260926 16:42:04.467397 30464 raft_consensus.cc:348] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.467885 30464 raft_consensus.cc:374] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:04.468022 30464 raft_consensus.cc:729] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b6866c56bdb47c1ad6e53b73d46339f, State: Initialized, Role: FOLLOWER
I20260926 16:42:04.468701 30464 consensus_queue.cc:260] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.470659 30464 ts_tablet_manager.cc:1428] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260926 16:42:04.473268 30465 raft_consensus.cc:348] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.473769 30465 raft_consensus.cc:374] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:04.473872 30465 raft_consensus.cc:729] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0cd491554d4298af2a40d8ac40288f, State: Initialized, Role: FOLLOWER
I20260926 16:42:04.474495 30465 consensus_queue.cc:260] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.474591 30466 raft_consensus.cc:348] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.474960 30466 raft_consensus.cc:374] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:04.475090 30466 raft_consensus.cc:729] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4bc497ccd743a5b0611ec5551fa00b, State: Initialized, Role: FOLLOWER
I20260926 16:42:04.475733 30445 heartbeater.cc:503] Master 127.10.110.62:43343 was elected leader, sending a full tablet report...
I20260926 16:42:04.475672 30466 consensus_queue.cc:260] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.476409 30465 ts_tablet_manager.cc:1428] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260926 16:42:04.477351 30311 heartbeater.cc:503] Master 127.10.110.62:43343 was elected leader, sending a full tablet report...
I20260926 16:42:04.477983 30466 ts_tablet_manager.cc:1428] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260926 16:42:04.487538 30178 tablet.cc:2369] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:04.512773 30472 raft_consensus.cc:482] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:04.513069 30472 raft_consensus.cc:504] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.514798 30472 leader_election.cc:288] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e0cd491554d4298af2a40d8ac40288f (127.10.110.3:42271), 8b6866c56bdb47c1ad6e53b73d46339f (127.10.110.1:41981)
I20260926 16:42:04.521597 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.515185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:46899 (local address 127.10.110.3:42271)
0926 16:42:04.515399 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.515406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.516410 (+ 1004us) server_negotiation.cc:407] Connection header received
0926 16:42:04.516615 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.516621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.516690 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.516765 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:04.517320 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.518055 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.519938 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.520690 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.520836 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.520914 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.521207 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.521336 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:42:04.521615 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.515390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:44027 (local address 127.10.110.1:41981)
0926 16:42:04.515619 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.515626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.516494 (+ 868us) server_negotiation.cc:407] Connection header received
0926 16:42:04.516678 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.516683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.516746 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.516820 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:04.517400 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.518096 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.519931 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.520705 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.520851 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.520927 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.521238 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.521386 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:42:04.521752 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.515060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:42271 (local address 127.10.110.2:46899)
0926 16:42:04.516302 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:04.516322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:04.516473 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:04.516793 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:04.516797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:04.516819 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:04.517186 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.517195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.518176 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.518181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.519776 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.519788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.520915 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.520924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.521062 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.521092 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:42:04.521172 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 16:42:04.521369 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":46,"thread_start_us":559,"threads_started":1}
I20260926 16:42:04.521857 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.515229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:41981 (local address 127.10.110.2:44027)
0926 16:42:04.516412 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:04.516430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:04.516540 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:04.516846 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:04.516850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:04.516876 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:04.517263 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.517272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.518230 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.518235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.519778 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:04.519789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.520940 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.520946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:04.521091 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.521113 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 16:42:04.521189 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 16:42:04.521313 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":22,"thread_start_us":451,"threads_started":1}
I20260926 16:42:04.522784 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e618d4961c34442a712b391978c209a" candidate_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" is_pre_election: true
I20260926 16:42:04.522770 30400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e618d4961c34442a712b391978c209a" candidate_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0cd491554d4298af2a40d8ac40288f" is_pre_election: true
I20260926 16:42:04.523226 30400 raft_consensus.cc:2454] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4bc497ccd743a5b0611ec5551fa00b in term 0.
I20260926 16:42:04.523239 30132 raft_consensus.cc:2454] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb4bc497ccd743a5b0611ec5551fa00b in term 0.
I20260926 16:42:04.523813 30200 leader_election.cc:302] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0cd491554d4298af2a40d8ac40288f, eb4bc497ccd743a5b0611ec5551fa00b; no voters:
I20260926 16:42:04.524235 30472 raft_consensus.cc:2781] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:04.524356 30472 raft_consensus.cc:482] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:04.524449 30472 raft_consensus.cc:3037] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:04.526628 30472 raft_consensus.cc:504] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.527222 30472 leader_election.cc:288] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [CANDIDATE]: Term 1 election: Requested vote from peers 6e0cd491554d4298af2a40d8ac40288f (127.10.110.3:42271), 8b6866c56bdb47c1ad6e53b73d46339f (127.10.110.1:41981)
I20260926 16:42:04.527710 30132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e618d4961c34442a712b391978c209a" candidate_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b6866c56bdb47c1ad6e53b73d46339f"
I20260926 16:42:04.527734 30400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e618d4961c34442a712b391978c209a" candidate_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0cd491554d4298af2a40d8ac40288f"
I20260926 16:42:04.527921 30132 raft_consensus.cc:3037] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:04.527949 30400 raft_consensus.cc:3037] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:04.530030 30132 raft_consensus.cc:2454] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4bc497ccd743a5b0611ec5551fa00b in term 1.
I20260926 16:42:04.530083 30400 raft_consensus.cc:2454] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb4bc497ccd743a5b0611ec5551fa00b in term 1.
I20260926 16:42:04.530458 30201 leader_election.cc:302] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b6866c56bdb47c1ad6e53b73d46339f, eb4bc497ccd743a5b0611ec5551fa00b; no voters:
I20260926 16:42:04.530774 30472 raft_consensus.cc:2781] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:04.531194 30472 raft_consensus.cc:686] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 1 LEADER]: Becoming Leader. State: Replica: eb4bc497ccd743a5b0611ec5551fa00b, State: Running, Role: LEADER
I20260926 16:42:04.531744 30472 consensus_queue.cc:237] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } }
I20260926 16:42:04.536451 29987 catalog_manager.cc:5495] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb4bc497ccd743a5b0611ec5551fa00b (127.10.110.2). New cstate: current_term: 1 leader_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb4bc497ccd743a5b0611ec5551fa00b" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 } health_report { overall_health: UNKNOWN } } }
W20260926 16:42:04.562114 30312 tablet.cc:2369] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:04.574602 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.569386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.10.110.2:45533)
0926 16:42:04.569609 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.569614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.569629 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:42:04.569735 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.569739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.569823 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.569974 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:04.570329 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.571070 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.572494 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.572889 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.573000 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.573480 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:04.574006 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:04.574137 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.574247 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.574331 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:42:04.582970 30132 raft_consensus.cc:1261] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 1 FOLLOWER]: Refusing update from remote peer eb4bc497ccd743a5b0611ec5551fa00b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:04.582970 30400 raft_consensus.cc:1261] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 1 FOLLOWER]: Refusing update from remote peer eb4bc497ccd743a5b0611ec5551fa00b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:04.583933 30475 consensus_queue.cc:1035] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b6866c56bdb47c1ad6e53b73d46339f" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:04.584281 30472 consensus_queue.cc:1035] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e0cd491554d4298af2a40d8ac40288f" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:42:04.586750 30446 tablet.cc:2369] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:04.599970 30478 mvcc.cc:205] Tried to move back new op lower bound from 7333646027082387456 to 7333646026886549504. Current Snapshot: MvccSnapshot[applied={T|T < 7333646027082387456 or (T in {7333646027082387456})}]
I20260926 16:42:04.613768 29987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:04.614082 29987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:04.618070 29987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3245dd33c6248d5b1ce9b34c10cdd91: Sending DeleteTablet for 3 replicas of tablet 2e618d4961c34442a712b391978c209a
I20260926 16:42:04.619376 30112 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e618d4961c34442a712b391978c209a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:04 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:37752
I20260926 16:42:04.619374 30380 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e618d4961c34442a712b391978c209a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:04 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:39830
I20260926 16:42:04.619987 30246 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e618d4961c34442a712b391978c209a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:04 UTC) from {username='kudu', principal='kudu/127.10.110.62@KRBTEST.COM'} at 127.0.0.1:49838
I20260926 16:42:04.621006 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30049
I20260926 16:42:04.632443 30490 tablet_replica.cc:330] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: stopping tablet replica
I20260926 16:42:04.633191 30490 raft_consensus.cc:2229] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:04.634157 30490 raft_consensus.cc:2258] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:04.636510 30490 ts_tablet_manager.cc:1905] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:04.644516 30490 ts_tablet_manager.cc:1918] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:04.644829 30490 log.cc:1196] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/2e618d4961c34442a712b391978c209a
I20260926 16:42:04.645485 30490 ts_tablet_manager.cc:1939] T 2e618d4961c34442a712b391978c209a P eb4bc497ccd743a5b0611ec5551fa00b: Deleting consensus metadata
I20260926 16:42:04.647586 29975 catalog_manager.cc:4841] TS eb4bc497ccd743a5b0611ec5551fa00b (127.10.110.2:45533): tablet 2e618d4961c34442a712b391978c209a (table test-table [id=f09f7e445dc34fa5bae6d951502ffeb9]) successfully deleted
I20260926 16:42:04.647921 30489 tablet_replica.cc:330] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: stopping tablet replica
I20260926 16:42:04.648322 30489 raft_consensus.cc:2229] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:04.648775 30489 raft_consensus.cc:2258] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:04.651144 30489 ts_tablet_manager.cc:1905] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:04.654320 30488 tablet_replica.cc:330] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: stopping tablet replica
I20260926 16:42:04.654714 30488 raft_consensus.cc:2229] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:04.655160 30488 raft_consensus.cc:2258] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:04.657903 30488 ts_tablet_manager.cc:1905] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:42:04 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.1@KRBTEST.COM
I20260926 16:42:04.658921 30489 ts_tablet_manager.cc:1918] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:04.659194 30489 log.cc:1196] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-0/wal/wals/2e618d4961c34442a712b391978c209a
I20260926 16:42:04.659819 30489 ts_tablet_manager.cc:1939] T 2e618d4961c34442a712b391978c209a P 8b6866c56bdb47c1ad6e53b73d46339f: Deleting consensus metadata
I20260926 16:42:04.662652 29974 catalog_manager.cc:4841] TS 8b6866c56bdb47c1ad6e53b73d46339f (127.10.110.1:41981): tablet 2e618d4961c34442a712b391978c209a (table test-table [id=f09f7e445dc34fa5bae6d951502ffeb9]) successfully deleted
I20260926 16:42:04.665184 30488 ts_tablet_manager.cc:1918] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:04.665365 30488 log.cc:1196] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/2e618d4961c34442a712b391978c209a
I20260926 16:42:04.665916 30488 ts_tablet_manager.cc:1939] T 2e618d4961c34442a712b391978c209a P 6e0cd491554d4298af2a40d8ac40288f: Deleting consensus metadata
I20260926 16:42:04.667388 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.622446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.10.110.1:41981)
0926 16:42:04.622650 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.622657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.644666 (+ 22009us) server_negotiation.cc:407] Connection header received
0926 16:42:04.645230 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.645237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.645340 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.645419 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:04.649181 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.650013 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.651983 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.652784 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.652910 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.659932 (+ 7022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:04.659947 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:04.659966 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:04.660020 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:04.663906 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.664875 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.664885 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.664888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.665081 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.665572 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.665577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.665581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.666113 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:04.666416 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.666775 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.667009 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:42:04.668114 29973 catalog_manager.cc:4841] TS 6e0cd491554d4298af2a40d8ac40288f (127.10.110.3:42271): tablet 2e618d4961c34442a712b391978c209a (table test-table [id=f09f7e445dc34fa5bae6d951502ffeb9]) successfully deleted
I20260926 16:42:04.668236 30172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:04.784564 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30049
I20260926 16:42:04.799219 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30183
Sep 26 16:42:04 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.2@KRBTEST.COM
I20260926 16:42:04.814649 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.799967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.10.110.2:45533)
0926 16:42:04.800158 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.800165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.800229 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:42:04.800407 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.800411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.800480 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.800644 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:04.801816 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.802620 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.803912 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.804371 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.804485 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.808791 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:04.808802 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:04.808815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:04.808858 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:04.811787 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.812504 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.812511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.812514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.812716 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.813123 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.813129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.813136 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.813491 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:04.813824 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.814101 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.814314 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:42:04.815284 30306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:04.929297 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30183
I20260926 16:42:04.944458 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30317
Sep 26 16:42:04 dist-test-slave-whjp krb5kdc[29943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.3@KRBTEST.COM
I20260926 16:42:04.958878 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:04.945117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.10.110.3:42271)
0926 16:42:04.945292 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:04.945298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:04.945337 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:42:04.945525 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:04.945530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:04.945610 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:04.945706 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:04.946981 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.947726 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.949019 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:04.949418 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:04.949527 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:04.953644 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:04.953658 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:04.953675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:04.953708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:04.956266 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.956912 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.956920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.956922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.957066 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:04.957412 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:04.957416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:04.957418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:04.957763 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:04.958032 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:04.958336 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 16:42:04.958566 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:42:04.959486 30440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:05.051584 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30317
I20260926 16:42:05.064944 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 29956
I20260926 16:42:05.066095 30018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:05.176898 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 29956
2026-09-26T16:42:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4588 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T16:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:05Z Disabled control of system clock
I20260926 16:42:05.247084 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39245
--webserver_interface=127.10.110.62
--webserver_port=0
--builtin_ntp_servers=127.10.110.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:42:05.570108 30499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:05.570429 30499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:05.570510 30499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:05.577831 30499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:05.577951 30499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:05.577988 30499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:05.578043 30499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:05.578086 30499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:05.588008 30499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:39245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:05.589344 30499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:05.591239 30499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:05.602533 30504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:05.602583 30505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:05.602617 30507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:05.603317 30499 server_base.cc:1034] running on GCE node
I20260926 16:42:05.604127 30499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:05.605497 30499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:05.606699 30499 hybrid_clock.cc:648] HybridClock initialized: now 1790440925606653 us; error 41 us; skew 500 ppm
I20260926 16:42:05.607028 30499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:42:05.609186 30499 webserver.cc:460] Webserver started at http://127.10.110.62:38343/ using document root <none> and password file <none>
I20260926 16:42:05.609824 30499 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:05.609917 30499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:05.610216 30499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:05.613078 30499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "84aa3e0d774a496498d1dec1b96817e1"
format_stamp: "Formatted at 2026-09-26 16:42:05 on dist-test-slave-whjp"
server_key: "fb1068a208223ecae4352dc497f93804"
server_key_iv: "788476107d72ae7d8ea70e0a9c075f90"
server_key_version: "encryptionkey@0"
I20260926 16:42:05.613858 30499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "84aa3e0d774a496498d1dec1b96817e1"
format_stamp: "Formatted at 2026-09-26 16:42:05 on dist-test-slave-whjp"
server_key: "fb1068a208223ecae4352dc497f93804"
server_key_iv: "788476107d72ae7d8ea70e0a9c075f90"
server_key_version: "encryptionkey@0"
I20260926 16:42:05.618942 30499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:05.622470 30513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:05.624089 30499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:42:05.624280 30499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "84aa3e0d774a496498d1dec1b96817e1"
format_stamp: "Formatted at 2026-09-26 16:42:05 on dist-test-slave-whjp"
server_key: "fb1068a208223ecae4352dc497f93804"
server_key_iv: "788476107d72ae7d8ea70e0a9c075f90"
server_key_version: "encryptionkey@0"
I20260926 16:42:05.624500 30499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:05.646582 30499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:05.647537 30499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:05.647869 30499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:05.667338 30499 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:39245
I20260926 16:42:05.667374 30564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:39245 every 8 connection(s)
I20260926 16:42:05.668772 30499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:05.670470 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30499
I20260926 16:42:05.670697 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:05.671233 10680 external_mini_cluster.cc:1428] Setting key d13a4288220814e0ce1f07eebdd3122e
I20260926 16:42:05.673698 30565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:05.685781 30565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1: Bootstrap starting.
I20260926 16:42:05.689458 30565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:05.690706 30565 log.cc:824] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:05.691408 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:05.673360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.10.110.62:39245)
0926 16:42:05.674484 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:05.674505 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:05.674553 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:42:05.676377 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:05.676404 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:05.676974 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:05.677434 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:05.678068 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:05.679718 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:05.681268 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:05.681995 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:05.682183 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:05.688684 (+ 6501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:05.688695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:05.688712 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:05.689305 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:05.689415 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:05.689459 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:05.689598 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:05.689858 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 16:42:05.690220 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":425,"threads_started":1}
I20260926 16:42:05.693961 30565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1: No bootstrap required, opened a new log
I20260926 16:42:05.701777 30565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aa3e0d774a496498d1dec1b96817e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39245 } }
I20260926 16:42:05.702203 30565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:05.702299 30565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aa3e0d774a496498d1dec1b96817e1, State: Initialized, Role: FOLLOWER
I20260926 16:42:05.702916 30565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aa3e0d774a496498d1dec1b96817e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39245 } }
I20260926 16:42:05.703147 30565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:05.703276 30565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:05.703439 30565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:05.705930 30565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aa3e0d774a496498d1dec1b96817e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39245 } }
I20260926 16:42:05.706453 30565 leader_election.cc:302] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84aa3e0d774a496498d1dec1b96817e1; no voters:
I20260926 16:42:05.707036 30565 leader_election.cc:288] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:05.707211 30570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:05.707968 30570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [term 1 LEADER]: Becoming Leader. State: Replica: 84aa3e0d774a496498d1dec1b96817e1, State: Running, Role: LEADER
I20260926 16:42:05.708487 30570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aa3e0d774a496498d1dec1b96817e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39245 } }
I20260926 16:42:05.709291 30565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:05.711618 30571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84aa3e0d774a496498d1dec1b96817e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aa3e0d774a496498d1dec1b96817e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39245 } } }
I20260926 16:42:05.711524 30572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84aa3e0d774a496498d1dec1b96817e1. Latest consensus state: current_term: 1 leader_uuid: "84aa3e0d774a496498d1dec1b96817e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84aa3e0d774a496498d1dec1b96817e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39245 } } }
I20260926 16:42:05.712208 30572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:05.712916 30571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:05.713693 30579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:05.719058 30579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:05.729229 30579 catalog_manager.cc:1347] Generated new cluster ID: 35bcd3aecb2b4ff3bb4fe1b9a063aec1
I20260926 16:42:05.729369 30579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:05.750218 30579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:05.751631 30579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:05.760607 30579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1: Generated new TSK 0
I20260926 16:42:05.761778 30579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:05.779815 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.1:0
--local_ip_for_outbound_sockets=127.10.110.1
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39245
--builtin_ntp_servers=127.10.110.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:42:06.164849 30589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:06.165205 30589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:06.165299 30589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:06.173082 30589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:06.173218 30589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:06.173328 30589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.1
I20260926 16:42:06.183076 30589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.110.1
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.1
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:06.184429 30589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:06.186260 30589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:06.201395 30595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.201354 30597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.201416 30594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:06.202322 30589 server_base.cc:1034] running on GCE node
I20260926 16:42:06.203011 30589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:06.204144 30589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:06.205386 30589 hybrid_clock.cc:648] HybridClock initialized: now 1790440926205326 us; error 57 us; skew 500 ppm
I20260926 16:42:06.205746 30589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:42:06.208355 30589 webserver.cc:460] Webserver started at http://127.10.110.1:38637/ using document root <none> and password file <none>
I20260926 16:42:06.209188 30589 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:06.209331 30589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:06.209750 30589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:06.214001 30589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data/instance:
uuid: "87325b9b3f524e598222f9687a5f0eb9"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-whjp"
server_key: "d40f99bfe881c4ee6766d84e5e42f4ba"
server_key_iv: "a0d4af5024eebf8d888c3f747cffdd77"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.215013 30589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance:
uuid: "87325b9b3f524e598222f9687a5f0eb9"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-whjp"
server_key: "d40f99bfe881c4ee6766d84e5e42f4ba"
server_key_iv: "a0d4af5024eebf8d888c3f747cffdd77"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.221396 30589 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:42:06.225581 30603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.227094 30589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 16:42:06.227293 30589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal
uuid: "87325b9b3f524e598222f9687a5f0eb9"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-whjp"
server_key: "d40f99bfe881c4ee6766d84e5e42f4ba"
server_key_iv: "a0d4af5024eebf8d888c3f747cffdd77"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.227531 30589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:06.247037 30589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:06.248062 30589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:06.248404 30589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:06.249531 30589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:06.251305 30589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:06.251425 30589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.251549 30589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:06.251624 30589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.284452 30589 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.1:34217
I20260926 16:42:06.284505 30715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.1:34217 every 8 connection(s)
I20260926 16:42:06.285977 30589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/data/info.pb
I20260926 16:42:06.294137 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30589
I20260926 16:42:06.294333 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-0/wal/instance
I20260926 16:42:06.294883 10680 external_mini_cluster.cc:1428] Setting key fe25b395c2abeec44d4cf2647468de90
I20260926 16:42:06.297830 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.2:0
--local_ip_for_outbound_sockets=127.10.110.2
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39245
--builtin_ntp_servers=127.10.110.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:42:06.301110 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.288920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:42471 (local address 127.10.110.62:39245)
0926 16:42:06.289448 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:06.289453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:06.291001 (+ 1548us) server_negotiation.cc:407] Connection header received
0926 16:42:06.293159 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:06.293166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:06.293269 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:06.293373 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:06.295180 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.296113 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.297887 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.298402 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.298524 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.300124 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:06.300130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:06.300133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:06.300157 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:06.300200 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:06.300215 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:06.300296 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:06.300550 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 16:42:06.300679 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
I20260926 16:42:06.302263 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.289613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39245 (local address 127.10.110.1:42471)
0926 16:42:06.290786 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:06.290835 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:06.292777 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:06.293697 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:06.293712 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:06.293862 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:06.294840 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.294885 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.296315 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.296320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.297701 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.297719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.298686 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.298695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.298941 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.299683 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:06.299685 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:06.299740 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:06.299886 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:06.300314 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:06.300328 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:06.300337 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:42:06.300518 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 16:42:06.301022 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":312,"threads_started":1}
I20260926 16:42:06.304354 30716 heartbeater.cc:347] Connected to a master server at 127.10.110.62:39245
I20260926 16:42:06.304710 30716 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:06.305593 30716 heartbeater.cc:511] Master 127.10.110.62:39245 requested a full tablet report, sending...
I20260926 16:42:06.307965 30530 ts_manager.cc:194] Registered new tserver with Master: 87325b9b3f524e598222f9687a5f0eb9 (127.10.110.1:34217)
I20260926 16:42:06.309564 30530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.1:42471
W20260926 16:42:06.646533 30720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:06.646860 30720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:06.646994 30720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:06.654553 30720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:06.654687 30720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:06.654850 30720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.2
I20260926 16:42:06.664206 30720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.110.2
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.2
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:06.665531 30720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:06.667292 30720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:06.683887 30725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.683945 30728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:06.683915 30726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:06.684458 30720 server_base.cc:1034] running on GCE node
I20260926 16:42:06.685163 30720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:06.686247 30720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:06.687482 30720 hybrid_clock.cc:648] HybridClock initialized: now 1790440926687444 us; error 30 us; skew 500 ppm
I20260926 16:42:06.687856 30720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:42:06.690333 30720 webserver.cc:460] Webserver started at http://127.10.110.2:42483/ using document root <none> and password file <none>
I20260926 16:42:06.691026 30720 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:06.691150 30720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:06.691540 30720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:06.694689 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data/instance:
uuid: "e77866d1dc3a4966be3fd3951bbd596a"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-whjp"
server_key: "b41f013f8653f2977752d0bb011d53ee"
server_key_iv: "7ba352139db48c8c3066d04c5a5a12ee"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.695500 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance:
uuid: "e77866d1dc3a4966be3fd3951bbd596a"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-whjp"
server_key: "b41f013f8653f2977752d0bb011d53ee"
server_key_iv: "7ba352139db48c8c3066d04c5a5a12ee"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.701068 30720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:42:06.704632 30734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.705996 30720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:06.706141 30720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal
uuid: "e77866d1dc3a4966be3fd3951bbd596a"
format_stamp: "Formatted at 2026-09-26 16:42:06 on dist-test-slave-whjp"
server_key: "b41f013f8653f2977752d0bb011d53ee"
server_key_iv: "7ba352139db48c8c3066d04c5a5a12ee"
server_key_version: "encryptionkey@0"
I20260926 16:42:06.706347 30720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:06.732721 30720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:06.733608 30720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:06.733907 30720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:06.734913 30720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:06.736502 30720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:06.736593 30720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.736708 30720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:06.736781 30720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:06.767314 30720 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.2:35499
I20260926 16:42:06.767372 30846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.2:35499 every 8 connection(s)
I20260926 16:42:06.768678 30720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/data/info.pb
I20260926 16:42:06.773685 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30720
I20260926 16:42:06.773846 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal/instance
I20260926 16:42:06.774617 10680 external_mini_cluster.cc:1428] Setting key 9e352b15ac79d8bd5d78fa912b3779c4
I20260926 16:42:06.778465 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.110.3:0
--local_ip_for_outbound_sockets=127.10.110.3
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39245
--builtin_ntp_servers=127.10.110.20:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:42:06.782851 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.771346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.2:54683 (local address 127.10.110.62:39245)
0926 16:42:06.771571 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:06.771578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:06.773046 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 16:42:06.775300 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:06.775306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:06.775408 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:06.775509 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:06.777154 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.778213 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.779968 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.780577 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.780695 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.782089 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:06.782095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:06.782099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:06.782122 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:06.782158 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:06.782172 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:06.782248 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:06.782446 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:42:06.782550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:42:06.783927 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:06.771942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39245 (local address 127.10.110.2:54683)
0926 16:42:06.772878 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:06.772916 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:06.774918 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:06.775936 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:06.775947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:06.776147 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:06.776913 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.776941 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.778394 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.778399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.779794 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:06.779809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:06.780823 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:06.780828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:06.781054 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:06.781734 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:06.781737 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:06.781779 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:06.781874 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:06.782257 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:06.782265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:06.782271 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:06.782410 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 16:42:06.782776 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":268,"threads_started":1}
I20260926 16:42:06.785370 30847 heartbeater.cc:347] Connected to a master server at 127.10.110.62:39245
I20260926 16:42:06.785705 30847 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:06.786466 30847 heartbeater.cc:511] Master 127.10.110.62:39245 requested a full tablet report, sending...
I20260926 16:42:06.787982 30530 ts_manager.cc:194] Registered new tserver with Master: e77866d1dc3a4966be3fd3951bbd596a (127.10.110.2:35499)
I20260926 16:42:06.789011 30530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.2:54683
W20260926 16:42:07.121783 30850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:07.122114 30850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:07.122212 30850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:07.129984 30850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:07.130115 30850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:07.130249 30850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.110.3
I20260926 16:42:07.141667 30850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.110.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.110.3
--webserver_port=0
--tserver_master_addrs=127.10.110.62:39245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.110.3
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:07.142989 30850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:07.145136 30850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:07.160545 30855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:07.160651 30856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:07.160646 30858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:07.161314 30850 server_base.cc:1034] running on GCE node
I20260926 16:42:07.162091 30850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:07.163255 30850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:07.164558 30850 hybrid_clock.cc:648] HybridClock initialized: now 1790440927164498 us; error 54 us; skew 500 ppm
I20260926 16:42:07.164927 30850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:42:07.167500 30850 webserver.cc:460] Webserver started at http://127.10.110.3:42091/ using document root <none> and password file <none>
I20260926 16:42:07.168378 30850 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:07.168515 30850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:07.168849 30850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:07.172045 30850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data/instance:
uuid: "2d59a1effdce4dd487196dbf309e1224"
format_stamp: "Formatted at 2026-09-26 16:42:07 on dist-test-slave-whjp"
server_key: "8b919b2cc59cc9f6fcd6cd2ca6c91b07"
server_key_iv: "c8f91b5933564a64e4ef1ed94c1de42f"
server_key_version: "encryptionkey@0"
I20260926 16:42:07.172927 30850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance:
uuid: "2d59a1effdce4dd487196dbf309e1224"
format_stamp: "Formatted at 2026-09-26 16:42:07 on dist-test-slave-whjp"
server_key: "8b919b2cc59cc9f6fcd6cd2ca6c91b07"
server_key_iv: "c8f91b5933564a64e4ef1ed94c1de42f"
server_key_version: "encryptionkey@0"
I20260926 16:42:07.178716 30850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:07.182713 30864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:07.184473 30850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:07.184674 30850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal
uuid: "2d59a1effdce4dd487196dbf309e1224"
format_stamp: "Formatted at 2026-09-26 16:42:07 on dist-test-slave-whjp"
server_key: "8b919b2cc59cc9f6fcd6cd2ca6c91b07"
server_key_iv: "c8f91b5933564a64e4ef1ed94c1de42f"
server_key_version: "encryptionkey@0"
I20260926 16:42:07.184909 30850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:07.208511 30850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:07.209566 30850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:07.209897 30850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:07.211004 30850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:42:07.212738 30850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:42:07.212913 30850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:07.213033 30850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:42:07.213120 30850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:07.244992 30850 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.3:44543
I20260926 16:42:07.245065 30976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.3:44543 every 8 connection(s)
I20260926 16:42:07.246301 30850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/data/info.pb
I20260926 16:42:07.255208 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 30850
I20260926 16:42:07.255371 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal/instance
I20260926 16:42:07.255812 10680 external_mini_cluster.cc:1428] Setting key a1bbb106efb6e3dcd6fce7068ce3312d
I20260926 16:42:07.259070 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.248923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.3:51235 (local address 127.10.110.62:39245)
0926 16:42:07.249146 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.249152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.250607 (+ 1455us) server_negotiation.cc:407] Connection header received
0926 16:42:07.252545 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.252550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.252634 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.252744 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.253866 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.254632 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.256329 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.256782 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.256928 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.258262 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.258269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.258273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:07.258296 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.258335 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:07.258351 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.258445 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.258683 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.258791 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:42:07.260196 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.249481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39245 (local address 127.10.110.3:51235)
0926 16:42:07.250413 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.250457 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.252202 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.252949 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.252957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.253061 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:07.253672 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.253691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.254776 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.254780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.256176 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.256189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.257010 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.257017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.257296 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.257937 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:42:07.257939 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:07.257982 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:42:07.258074 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:07.258460 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:07.258467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:07.258473 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.258684 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.259068 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":272,"threads_started":1}
I20260926 16:42:07.261669 30977 heartbeater.cc:347] Connected to a master server at 127.10.110.62:39245
I20260926 16:42:07.261961 30977 heartbeater.cc:464] Registering TS with master...
I20260926 16:42:07.262730 30977 heartbeater.cc:511] Master 127.10.110.62:39245 requested a full tablet report, sending...
I20260926 16:42:07.264269 30530 ts_manager.cc:194] Registered new tserver with Master: 2d59a1effdce4dd487196dbf309e1224 (127.10.110.3:44543)
I20260926 16:42:07.265142 30530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.110.3:51235
I20260926 16:42:07.271543 10680 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:42:07.283808 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.277629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.10.110.62:39245)
0926 16:42:07.277820 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.277825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.278273 (+ 448us) server_negotiation.cc:407] Connection header received
0926 16:42:07.278448 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.278453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.278537 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.278627 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.279217 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.279991 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.281388 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.281834 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.281934 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.282975 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.282979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.282983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:07.283005 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.283063 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:07.283083 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.283165 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.283374 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.283462 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:42:07.288455 30530 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:42:07.295521 30530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:42:07.313079 30716 heartbeater.cc:503] Master 127.10.110.62:39245 was elected leader, sending a full tablet report...
I20260926 16:42:07.331969 30989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.310414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:35499 (local address 127.0.0.1:59880)
0926 16:42:07.322445 (+ 12031us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.322465 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.322623 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.323677 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.323681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.323720 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:07.324257 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.324278 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.326128 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.326133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.328477 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.328491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.330409 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.330415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.330647 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.330725 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.330865 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.331101 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11920,"mutex_wait_us":185,"spinlock_wait_cycles":14976,"thread_start_us":10116,"threads_started":1}
I20260926 16:42:07.332104 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.310886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.10.110.2:35499)
0926 16:42:07.321940 (+ 11054us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.321949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.322766 (+ 817us) server_negotiation.cc:407] Connection header received
0926 16:42:07.322883 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.322892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.323182 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.323395 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:07.324470 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.325965 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.328699 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.330197 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.330346 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.330633 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.330986 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.331525 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10892,"thread_start_us":10784,"threads_started":1}
I20260926 16:42:07.332969 30991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.309583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59668)
0926 16:42:07.322445 (+ 12862us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.322465 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.322623 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.323669 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.323677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.323718 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:07.324592 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.324606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.326401 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.326406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.329077 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.329094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.331732 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.331741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.331975 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.332013 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.332197 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.332355 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12715,"mutex_wait_us":431,"spinlock_wait_cycles":16512,"thread_start_us":12534,"threads_started":1}
I20260926 16:42:07.334149 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.309382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44543 (local address 127.0.0.1:41566)
0926 16:42:07.321110 (+ 11728us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.321169 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.321443 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.325142 (+ 3699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.325148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.325187 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:07.325570 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.325582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.327498 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.327504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.329674 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.329688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.332965 (+ 3277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.332974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.333290 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.333389 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.333522 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.333762 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11344,"thread_start_us":13464,"threads_started":1}
I20260926 16:42:07.334149 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.309893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.10.110.1:34217)
0926 16:42:07.310650 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.310661 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.322771 (+ 12110us) server_negotiation.cc:407] Connection header received
0926 16:42:07.322883 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.322892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.323191 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.323395 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:07.324800 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.326269 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.329336 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.330917 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.331200 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.331516 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.333374 (+ 1858us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.333764 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":1138,"threads_started":1}
I20260926 16:42:07.335778 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.309604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.10.110.3:44543)
0926 16:42:07.324271 (+ 14667us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.324280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.324316 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:42:07.324407 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.324416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.324671 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.324867 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:07.325776 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.327333 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.329916 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.332366 (+ 2450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.334575 (+ 2209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.334958 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.335038 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.335385 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14464,"thread_start_us":14309,"threads_started":1}
I20260926 16:42:07.338438 30782 tablet_service.cc:1467] Processing CreateTablet for tablet e395657586974e86a1c1b704480b1fa3 (DEFAULT_TABLE table=test-table [id=5d4b71e51bd04c408db4cd6dbb759b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:07.338428 30651 tablet_service.cc:1467] Processing CreateTablet for tablet e395657586974e86a1c1b704480b1fa3 (DEFAULT_TABLE table=test-table [id=5d4b71e51bd04c408db4cd6dbb759b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:07.339789 30782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e395657586974e86a1c1b704480b1fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:07.339926 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e395657586974e86a1c1b704480b1fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:07.340080 30912 tablet_service.cc:1467] Processing CreateTablet for tablet e395657586974e86a1c1b704480b1fa3 (DEFAULT_TABLE table=test-table [id=5d4b71e51bd04c408db4cd6dbb759b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:42:07.341673 30912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e395657586974e86a1c1b704480b1fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:07.355204 30995 tablet_bootstrap.cc:492] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Bootstrap starting.
I20260926 16:42:07.358775 30997 tablet_bootstrap.cc:492] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Bootstrap starting.
I20260926 16:42:07.359282 30995 tablet_bootstrap.cc:654] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:07.360664 30996 tablet_bootstrap.cc:492] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Bootstrap starting.
I20260926 16:42:07.360865 30995 log.cc:824] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:07.362541 30997 tablet_bootstrap.cc:654] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:07.363888 30997 log.cc:824] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:07.363993 30996 tablet_bootstrap.cc:654] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:07.364777 30995 tablet_bootstrap.cc:492] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: No bootstrap required, opened a new log
I20260926 16:42:07.365090 30995 ts_tablet_manager.cc:1397] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260926 16:42:07.365679 30996 log.cc:824] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:07.368434 30997 tablet_bootstrap.cc:492] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: No bootstrap required, opened a new log
I20260926 16:42:07.368746 30997 ts_tablet_manager.cc:1397] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:42:07.370833 30996 tablet_bootstrap.cc:492] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: No bootstrap required, opened a new log
I20260926 16:42:07.371169 30996 ts_tablet_manager.cc:1397] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 16:42:07.373410 30995 raft_consensus.cc:348] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.373847 30995 raft_consensus.cc:374] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:07.373942 30995 raft_consensus.cc:729] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77866d1dc3a4966be3fd3951bbd596a, State: Initialized, Role: FOLLOWER
I20260926 16:42:07.374603 30995 consensus_queue.cc:260] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.375953 30847 heartbeater.cc:503] Master 127.10.110.62:39245 was elected leader, sending a full tablet report...
I20260926 16:42:07.376531 30995 ts_tablet_manager.cc:1428] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260926 16:42:07.380313 30997 raft_consensus.cc:348] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.380887 30997 raft_consensus.cc:374] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:07.381006 30997 raft_consensus.cc:729] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d59a1effdce4dd487196dbf309e1224, State: Initialized, Role: FOLLOWER
I20260926 16:42:07.381800 30997 consensus_queue.cc:260] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.382917 30977 heartbeater.cc:503] Master 127.10.110.62:39245 was elected leader, sending a full tablet report...
I20260926 16:42:07.382670 30996 raft_consensus.cc:348] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.383175 30996 raft_consensus.cc:374] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:07.383294 30996 raft_consensus.cc:729] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87325b9b3f524e598222f9687a5f0eb9, State: Initialized, Role: FOLLOWER
I20260926 16:42:07.383812 30997 ts_tablet_manager.cc:1428] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260926 16:42:07.383989 30996 consensus_queue.cc:260] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.385874 30996 ts_tablet_manager.cc:1428] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260926 16:42:07.497934 30978 tablet.cc:2369] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:07.520815 30848 tablet.cc:2369] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:07.538511 30717 tablet.cc:2369] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:07.591499 31003 raft_consensus.cc:482] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:42:07.591800 31003 raft_consensus.cc:504] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.593593 31003 leader_election.cc:288] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d59a1effdce4dd487196dbf309e1224 (127.10.110.3:44543), e77866d1dc3a4966be3fd3951bbd596a (127.10.110.2:35499)
I20260926 16:42:07.603001 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.594229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:48555 (local address 127.10.110.2:35499)
0926 16:42:07.594433 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.594440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.595356 (+ 916us) server_negotiation.cc:407] Connection header received
0926 16:42:07.595548 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.595553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.595643 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.595743 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:07.596592 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.597686 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.600748 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.601906 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.602083 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.602157 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.602634 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.602756 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:42:07.603001 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.594200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.110.1:58143 (local address 127.10.110.3:44543)
0926 16:42:07.594433 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.594439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.595337 (+ 898us) server_negotiation.cc:407] Connection header received
0926 16:42:07.595559 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.595564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.595643 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.595744 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:42:07.596652 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.597736 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.600748 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.601966 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.602099 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.602157 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.602578 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.602756 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:42:07.603175 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.594121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.2:35499 (local address 127.10.110.1:48555)
0926 16:42:07.595260 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.595273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.595398 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.595776 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.595782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.595849 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:07.596429 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.596444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.597869 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.597876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.600536 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.600545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.602132 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.602139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.602393 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.602449 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.602599 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.602703 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":264,"spinlock_wait_cycles":15488,"thread_start_us":578,"threads_started":1}
I20260926 16:42:07.603189 31004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.594090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.3:44543 (local address 127.10.110.1:58143)
0926 16:42:07.595197 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.595217 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:07.595443 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:07.595777 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:07.595782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:07.595852 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:42:07.596436 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.596445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.597870 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.597876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.600519 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:07.600536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.602135 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.602140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:07.602398 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.602445 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 16:42:07.602542 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 16:42:07.602686 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"mutex_wait_us":236,"spinlock_wait_cycles":49536,"thread_start_us":597,"threads_started":1}
I20260926 16:42:07.604426 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e395657586974e86a1c1b704480b1fa3" candidate_uuid: "87325b9b3f524e598222f9687a5f0eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d59a1effdce4dd487196dbf309e1224" is_pre_election: true
I20260926 16:42:07.604437 30802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e395657586974e86a1c1b704480b1fa3" candidate_uuid: "87325b9b3f524e598222f9687a5f0eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77866d1dc3a4966be3fd3951bbd596a" is_pre_election: true
I20260926 16:42:07.605124 30932 raft_consensus.cc:2454] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87325b9b3f524e598222f9687a5f0eb9 in term 0.
I20260926 16:42:07.605124 30802 raft_consensus.cc:2454] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87325b9b3f524e598222f9687a5f0eb9 in term 0.
I20260926 16:42:07.605866 30604 leader_election.cc:302] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87325b9b3f524e598222f9687a5f0eb9, e77866d1dc3a4966be3fd3951bbd596a; no voters:
I20260926 16:42:07.606348 31003 raft_consensus.cc:2781] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:42:07.606484 31003 raft_consensus.cc:482] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:42:07.606582 31003 raft_consensus.cc:3037] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:07.608846 31003 raft_consensus.cc:504] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.609459 31003 leader_election.cc:288] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d59a1effdce4dd487196dbf309e1224 (127.10.110.3:44543), e77866d1dc3a4966be3fd3951bbd596a (127.10.110.2:35499)
I20260926 16:42:07.610060 30802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e395657586974e86a1c1b704480b1fa3" candidate_uuid: "87325b9b3f524e598222f9687a5f0eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e77866d1dc3a4966be3fd3951bbd596a"
I20260926 16:42:07.610059 30932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e395657586974e86a1c1b704480b1fa3" candidate_uuid: "87325b9b3f524e598222f9687a5f0eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d59a1effdce4dd487196dbf309e1224"
I20260926 16:42:07.610306 30932 raft_consensus.cc:3037] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:07.610313 30802 raft_consensus.cc:3037] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:07.613538 30932 raft_consensus.cc:2454] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87325b9b3f524e598222f9687a5f0eb9 in term 1.
I20260926 16:42:07.613564 30802 raft_consensus.cc:2454] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87325b9b3f524e598222f9687a5f0eb9 in term 1.
I20260926 16:42:07.614025 30605 leader_election.cc:302] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d59a1effdce4dd487196dbf309e1224, 87325b9b3f524e598222f9687a5f0eb9; no voters:
I20260926 16:42:07.614370 31003 raft_consensus.cc:2781] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:07.614853 31003 raft_consensus.cc:686] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 1 LEADER]: Becoming Leader. State: Replica: 87325b9b3f524e598222f9687a5f0eb9, State: Running, Role: LEADER
I20260926 16:42:07.615340 31003 consensus_queue.cc:237] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } }
I20260926 16:42:07.619796 30530 catalog_manager.cc:5495] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87325b9b3f524e598222f9687a5f0eb9 (127.10.110.1). New cstate: current_term: 1 leader_uuid: "87325b9b3f524e598222f9687a5f0eb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87325b9b3f524e598222f9687a5f0eb9" member_type: VOTER last_known_addr { host: "127.10.110.1" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 } health_report { overall_health: UNKNOWN } } }
I20260926 16:42:07.705962 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.700155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.10.110.1:34217)
0926 16:42:07.700333 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.700338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.700460 (+ 122us) server_negotiation.cc:407] Connection header received
0926 16:42:07.700660 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.700664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.700760 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.700891 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:07.701181 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.701856 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.703287 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.703872 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.704098 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.704705 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:07.705284 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:07.705423 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.705672 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.705749 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:42:07.717976 30932 raft_consensus.cc:1261] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 1 FOLLOWER]: Refusing update from remote peer 87325b9b3f524e598222f9687a5f0eb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:07.717976 30802 raft_consensus.cc:1261] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 1 FOLLOWER]: Refusing update from remote peer 87325b9b3f524e598222f9687a5f0eb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:07.718989 31006 consensus_queue.cc:1035] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e77866d1dc3a4966be3fd3951bbd596a" member_type: VOTER last_known_addr { host: "127.10.110.2" port: 35499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:07.719375 31003 consensus_queue.cc:1035] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d59a1effdce4dd487196dbf309e1224" member_type: VOTER last_known_addr { host: "127.10.110.3" port: 44543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:07.731176 31009 mvcc.cc:205] Tried to move back new op lower bound from 7333646039922352128 to 7333646039516110848. Current Snapshot: MvccSnapshot[applied={T|T < 7333646039922352128}]
I20260926 16:42:07.732478 31011 mvcc.cc:205] Tried to move back new op lower bound from 7333646039922352128 to 7333646039516110848. Current Snapshot: MvccSnapshot[applied={T|T < 7333646039922352128}]
I20260926 16:42:07.750378 30529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:07.750761 30529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:42:07.755239 30529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84aa3e0d774a496498d1dec1b96817e1: Sending DeleteTablet for 3 replicas of tablet e395657586974e86a1c1b704480b1fa3
I20260926 16:42:07.756659 30651 tablet_service.cc:1514] Processing DeleteTablet for tablet e395657586974e86a1c1b704480b1fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:07 UTC) from {username='slave'} at 127.0.0.1:59668
I20260926 16:42:07.756683 30912 tablet_service.cc:1514] Processing DeleteTablet for tablet e395657586974e86a1c1b704480b1fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:07 UTC) from {username='slave'} at 127.0.0.1:41566
I20260926 16:42:07.756690 30782 tablet_service.cc:1514] Processing DeleteTablet for tablet e395657586974e86a1c1b704480b1fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:42:07 UTC) from {username='slave'} at 127.0.0.1:59880
I20260926 16:42:07.761135 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30589
I20260926 16:42:07.764355 31021 tablet_replica.cc:330] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: stopping tablet replica
I20260926 16:42:07.764916 31021 raft_consensus.cc:2229] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:07.765487 31021 raft_consensus.cc:2258] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:07.767946 31021 ts_tablet_manager.cc:1905] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:07.771654 31019 tablet_replica.cc:330] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: stopping tablet replica
I20260926 16:42:07.772415 31019 raft_consensus.cc:2229] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:42:07.773274 31019 raft_consensus.cc:2258] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:07.774130 31020 tablet_replica.cc:330] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: stopping tablet replica
I20260926 16:42:07.774633 31020 raft_consensus.cc:2229] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:42:07.775684 31021 ts_tablet_manager.cc:1918] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:07.775894 31021 log.cc:1196] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-1/wal/wals/e395657586974e86a1c1b704480b1fa3
I20260926 16:42:07.776077 31020 raft_consensus.cc:2258] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:42:07.776422 31021 ts_tablet_manager.cc:1939] T e395657586974e86a1c1b704480b1fa3 P e77866d1dc3a4966be3fd3951bbd596a: Deleting consensus metadata
I20260926 16:42:07.777469 31019 ts_tablet_manager.cc:1905] T e395657586974e86a1c1b704480b1fa3 P 87325b9b3f524e598222f9687a5f0eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:07.778962 30992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.762439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.10.110.1:34217)
0926 16:42:07.762653 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.762660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.765673 (+ 3013us) server_negotiation.cc:407] Connection header received
0926 16:42:07.766614 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.766621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.766765 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.766865 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.768271 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.769577 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.773637 (+ 4060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.774277 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.774387 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.777019 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.777033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.777051 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:07.777917 (+ 866us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.777984 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:07.778022 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.778147 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.778418 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.778584 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:42:07.779393 30514 catalog_manager.cc:4841] TS e77866d1dc3a4966be3fd3951bbd596a (127.10.110.2:35499): tablet e395657586974e86a1c1b704480b1fa3 (table test-table [id=5d4b71e51bd04c408db4cd6dbb759b0f]) successfully deleted
I20260926 16:42:07.780171 30711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:07.782075 31020 ts_tablet_manager.cc:1905] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:42:07.788791 31020 ts_tablet_manager.cc:1918] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:42:07.788969 31020 log.cc:1196] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Deleting WAL directory at /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440615325743-10680-0/minicluster-data/ts-2/wal/wals/e395657586974e86a1c1b704480b1fa3
I20260926 16:42:07.789381 31020 ts_tablet_manager.cc:1939] T e395657586974e86a1c1b704480b1fa3 P 2d59a1effdce4dd487196dbf309e1224: Deleting consensus metadata
I20260926 16:42:07.791144 30515 catalog_manager.cc:4841] TS 2d59a1effdce4dd487196dbf309e1224 (127.10.110.3:44543): tablet e395657586974e86a1c1b704480b1fa3 (table test-table [id=5d4b71e51bd04c408db4cd6dbb759b0f]) successfully deleted
I20260926 16:42:07.884804 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30589
W20260926 16:42:07.900202 30516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.110.1:34217 (error 108)
I20260926 16:42:07.900902 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30720
I20260926 16:42:07.902346 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.901491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59700)
0926 16:42:07.902057 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.902179 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":360,"threads_started":1}
W20260926 16:42:07.902580 30516 catalog_manager.cc:4584] TS 87325b9b3f524e598222f9687a5f0eb9 (127.10.110.1:34217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e395657586974e86a1c1b704480b1fa3: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
I20260926 16:42:07.907732 30994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.901727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.10.110.2:35499)
0926 16:42:07.901904 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:07.901910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:07.901930 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:42:07.902028 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:07.902034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:07.902128 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:07.902214 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:07.902680 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.903446 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.904806 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:07.905257 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:07.905359 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:07.906301 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:07.906310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:07.906325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:07.906933 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:07.906980 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:07.907000 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:07.907091 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:07.907289 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 16:42:07.907390 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:42:07.908494 30842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:07.956966 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.956405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59714)
0926 16:42:07.956663 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.956775 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 16:42:07.958235 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:07.957799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59720)
0926 16:42:07.957972 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:07.958065 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 16:42:08.013325 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30720
I20260926 16:42:08.018818 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.018259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59726)
0926 16:42:08.018498 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:08.018622 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:42:08.020121 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.019631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59730)
0926 16:42:08.019786 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:08.019925 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 16:42:08.028998 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30850
I20260926 16:42:08.035728 30990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.029854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.10.110.3:44543)
0926 16:42:08.030042 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:08.030048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:08.030069 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:42:08.030165 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:08.030170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:08.030282 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:08.030369 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:08.030806 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.031511 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.032788 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:08.033191 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:08.033304 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:08.034367 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:08.034376 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:08.034392 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:08.034970 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:08.035024 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:08.035045 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:08.035144 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:08.035284 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 16:42:08.035398 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:42:08.036392 30972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:08.100891 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.100380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59744)
0926 16:42:08.100627 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:08.100720 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 16:42:08.102360 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:08.101732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.1:34217 (local address 127.0.0.1:59748)
0926 16:42:08.102055 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:08.102197 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.1:34217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260926 16:42:08.131948 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30850
I20260926 16:42:08.146957 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 30499
I20260926 16:42:08.148276 30560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:08.265695 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 30499
2026-09-26T16:42:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3090 ms)
[----------] 6 tests from EncryptionPolicyTest (22986 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:08 dist-test-slave-whjp krb5kdc[31031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:08 dist-test-slave-whjp krb5kdc[31031](info): set up 2 sockets
Sep 26 16:42:08 dist-test-slave-whjp krb5kdc[31031](info): commencing operation
krb5kdc: starting...
W20260926 16:42:09.341005 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:09 dist-test-slave-whjp krb5kdc[31031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:09.616024 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:09.967718 31044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:09.968210 31044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:09.968393 31044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:09.976104 31044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:09.976257 31044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:09.976305 31044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:09.976365 31044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:09.976408 31044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:09.986991 31044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:09.988476 31044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:09.989842 31044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:10.003386 31050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:10.003515 31052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:10.003366 31049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:10.004770 31044 server_base.cc:1034] running on GCE node
I20260926 16:42:10.005725 31044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:10.007437 31044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:10.008669 31044 hybrid_clock.cc:648] HybridClock initialized: now 1790440930008624 us; error 41 us; skew 500 ppm
I20260926 16:42:10.009063 31044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:10 dist-test-slave-whjp krb5kdc[31031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:10.012727 31044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:10.014866 31044 webserver.cc:460] Webserver started at http://127.0.0.1:44907/ using document root <none> and password file <none>
I20260926 16:42:10.015662 31044 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:10.015813 31044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:10.016242 31044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:10.019681 31044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "cd80d84f22d34d3a9c14a38404dc8ef5"
format_stamp: "Formatted at 2026-09-26 16:42:10 on dist-test-slave-whjp"
server_key: "ec84293e430b26b70a04255996752fe7"
server_key_iv: "8f8f587d25dc4286ca303856667bd885"
server_key_version: "encryptionkey@0"
I20260926 16:42:10.020634 31044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "cd80d84f22d34d3a9c14a38404dc8ef5"
format_stamp: "Formatted at 2026-09-26 16:42:10 on dist-test-slave-whjp"
server_key: "ec84293e430b26b70a04255996752fe7"
server_key_iv: "8f8f587d25dc4286ca303856667bd885"
server_key_version: "encryptionkey@0"
I20260926 16:42:10.026901 31044 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:10.031137 31059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:10.032795 31044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:10.032997 31044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "cd80d84f22d34d3a9c14a38404dc8ef5"
format_stamp: "Formatted at 2026-09-26 16:42:10 on dist-test-slave-whjp"
server_key: "ec84293e430b26b70a04255996752fe7"
server_key_iv: "8f8f587d25dc4286ca303856667bd885"
server_key_version: "encryptionkey@0"
I20260926 16:42:10.033228 31044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:10.051403 31044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:10.056360 31044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:10.056674 31044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:10.077121 31044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37665
I20260926 16:42:10.077306 31110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37665 every 8 connection(s)
I20260926 16:42:10.078619 31044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:10.082690 31111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:10.082969 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31044
I20260926 16:42:10.083153 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:10.083868 10680 external_mini_cluster.cc:1428] Setting key c6ae031469210c9d202e0f73bc5f05cd
I20260926 16:42:10.098884 31111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5: Bootstrap starting.
Sep 26 16:42:10 dist-test-slave-whjp krb5kdc[31031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:42:10.102805 31111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:10.104290 31111 log.cc:824] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:10.108145 31111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5: No bootstrap required, opened a new log
I20260926 16:42:10.110589 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:10.085857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53825 (local address 127.0.0.1:37665)
0926 16:42:10.086822 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:10.086838 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:10.086884 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:42:10.088530 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:10.088559 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:10.089081 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:10.089495 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:10.091532 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:10.093027 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:10.094957 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:10.095941 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:10.096118 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:10.101064 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:10.101077 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:10.101098 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:10.101170 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:10.105134 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:10.106387 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:10.106397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:10.106407 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:10.106674 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:10.107139 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:10.107144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:10.107148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:10.107895 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:10.108223 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:10.108756 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 16:42:10.109164 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":429,"threads_started":1}
I20260926 16:42:10.115469 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:10.117718 31111 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260926 16:42:10.118144 31111 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:10.118242 31111 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd80d84f22d34d3a9c14a38404dc8ef5, State: Initialized, Role: FOLLOWER
I20260926 16:42:10.118944 31111 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260926 16:42:10.119199 31111 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:10.119333 31111 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:10.119491 31111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:10.122282 31111 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260926 16:42:10.122850 31111 leader_election.cc:302] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd80d84f22d34d3a9c14a38404dc8ef5; no voters:
I20260926 16:42:10.123598 31111 leader_election.cc:288] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:10.123728 31121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:10.125939 31121 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [term 1 LEADER]: Becoming Leader. State: Replica: cd80d84f22d34d3a9c14a38404dc8ef5, State: Running, Role: LEADER
I20260926 16:42:10.126084 31111 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:10.126541 31121 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } }
I20260926 16:42:10.132436 31123 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd80d84f22d34d3a9c14a38404dc8ef5. Latest consensus state: current_term: 1 leader_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } } }
I20260926 16:42:10.132908 31123 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:10.133704 31122 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd80d84f22d34d3a9c14a38404dc8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37665 } } }
I20260926 16:42:10.134429 31122 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:10.142392 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:10.120452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53129 (local address 127.0.0.1:37665)
0926 16:42:10.120664 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:10.120674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:10.121059 (+ 385us) server_negotiation.cc:407] Connection header received
0926 16:42:10.121218 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:10.121224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:10.121304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:10.121420 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:10.123012 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:10.124256 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:10.126155 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:10.126831 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:10.126982 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:10.130877 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:10.130886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:10.130892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:10.130947 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:10.136527 (+ 5580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:10.138343 (+ 1816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:10.138349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:10.138354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:10.138562 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:10.139271 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:10.139277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:10.139281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:10.139780 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:10.141487 (+ 1707us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:10.141702 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 16:42:10.141946 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:42:10.145931 31130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:10.151306 31130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:10.161372 31130 catalog_manager.cc:1347] Generated new cluster ID: fc480c40fbf8412eae80f3bc3a8b845c
I20260926 16:42:10.161512 31130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:10.187609 31130 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:10.189699 31130 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:10.208560 31130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd80d84f22d34d3a9c14a38404dc8ef5: Generated new TSK 0
I20260926 16:42:10.209753 31130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:10.221032 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31044
I20260926 16:42:10.222503 31106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:10.319651 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31044
W20260926 16:42:10.335458 31029 connection.cc:535] client connection to 127.0.0.1:37665 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37665): Connection reset by peer (error 104)
2026-09-26T16:42:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:10 dist-test-slave-whjp krb5kdc[31143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:10 dist-test-slave-whjp krb5kdc[31143](info): set up 2 sockets
Sep 26 16:42:10 dist-test-slave-whjp krb5kdc[31143](info): commencing operation
krb5kdc: starting...
W20260926 16:42:11.368970 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:11 dist-test-slave-whjp krb5kdc[31143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:11.636320 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:11.988504 31156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:11.988826 31156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:11.988951 31156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:11.996455 31156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:11.996599 31156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:11.996673 31156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:11.996757 31156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:11.996810 31156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:12.007237 31156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:12.008687 31156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:12.010304 31156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:12.022981 31161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:12.023022 31162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:12.022992 31164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:12.024127 31156 server_base.cc:1034] running on GCE node
I20260926 16:42:12.024977 31156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:12.026608 31156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:12.027889 31156 hybrid_clock.cc:648] HybridClock initialized: now 1790440932027797 us; error 64 us; skew 500 ppm
I20260926 16:42:12.028337 31156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:12 dist-test-slave-whjp krb5kdc[31143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:12.038561 31156 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:12.040387 31156 webserver.cc:460] Webserver started at http://127.0.0.1:38157/ using document root <none> and password file <none>
I20260926 16:42:12.041193 31156 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:12.041326 31156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:12.041606 31156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:12.044895 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "fe956765e5134132ae6b9be348e69271"
format_stamp: "Formatted at 2026-09-26 16:42:12 on dist-test-slave-whjp"
server_key: "3bbcfcf5002bc01fddc3c5797e66fbfc"
server_key_iv: "e102b87d4990db624b87e26868d64289"
server_key_version: "encryptionkey@0"
I20260926 16:42:12.045768 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "fe956765e5134132ae6b9be348e69271"
format_stamp: "Formatted at 2026-09-26 16:42:12 on dist-test-slave-whjp"
server_key: "3bbcfcf5002bc01fddc3c5797e66fbfc"
server_key_iv: "e102b87d4990db624b87e26868d64289"
server_key_version: "encryptionkey@0"
I20260926 16:42:12.051402 31156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 16:42:12.055441 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:12.057426 31156 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:42:12.057649 31156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "fe956765e5134132ae6b9be348e69271"
format_stamp: "Formatted at 2026-09-26 16:42:12 on dist-test-slave-whjp"
server_key: "3bbcfcf5002bc01fddc3c5797e66fbfc"
server_key_iv: "e102b87d4990db624b87e26868d64289"
server_key_version: "encryptionkey@0"
I20260926 16:42:12.057876 31156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:12.082885 31156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:12.087298 31156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:12.087563 31156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:12.105669 31156 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37871
I20260926 16:42:12.105670 31222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37871 every 8 connection(s)
I20260926 16:42:12.107118 31156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:12.111078 31223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:12.112805 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31156
I20260926 16:42:12.112967 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:12.113421 10680 external_mini_cluster.cc:1428] Setting key 1196d6df2a01ea35f7e9ef53544cd1d6
I20260926 16:42:12.125376 31223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271: Bootstrap starting.
I20260926 16:42:12.126976 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.115181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32973 (local address 127.0.0.1:37871)
0926 16:42:12.116013 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.116024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.116071 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:42:12.117676 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.117701 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.118150 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.118488 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:12.119139 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.120511 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.122414 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.123185 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.123323 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.124664 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:12.124673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:12.124689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:12.125242 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:12.125321 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:12.125346 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:12.125467 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.125743 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.125936 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":375,"threads_started":1}
I20260926 16:42:12.128731 31223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:12.129841 31223 log.cc:824] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:12.130858 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:12.133049 31223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271: No bootstrap required, opened a new log
I20260926 16:42:12.142230 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:12.135263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40475 (local address 127.0.0.1:37871)
0926 16:42:12.135456 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:12.135466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:12.135862 (+ 396us) server_negotiation.cc:407] Connection header received
0926 16:42:12.136076 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:12.136082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:12.136160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:12.136283 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:12.136908 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.138070 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.139534 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:12.140276 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:12.140427 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:12.141489 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:12.141495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:12.141500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:12.141523 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:12.141586 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:12.141607 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:12.141712 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:12.141850 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 16:42:12.141906 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:42:12.142499 31223 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe956765e5134132ae6b9be348e69271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260926 16:42:12.142939 31223 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:12.143061 31223 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe956765e5134132ae6b9be348e69271, State: Initialized, Role: FOLLOWER
I20260926 16:42:12.143718 31223 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe956765e5134132ae6b9be348e69271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260926 16:42:12.144024 31223 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:12.144156 31223 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:12.144286 31223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:12.146394 31223 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe956765e5134132ae6b9be348e69271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260926 16:42:12.146888 31223 leader_election.cc:302] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe956765e5134132ae6b9be348e69271; no voters:
I20260926 16:42:12.147446 31223 leader_election.cc:288] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:12.147584 31233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:12.148695 31233 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [term 1 LEADER]: Becoming Leader. State: Replica: fe956765e5134132ae6b9be348e69271, State: Running, Role: LEADER
I20260926 16:42:12.149319 31233 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe956765e5134132ae6b9be348e69271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } }
I20260926 16:42:12.149745 31223 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:12.152204 31234 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe956765e5134132ae6b9be348e69271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe956765e5134132ae6b9be348e69271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } } }
I20260926 16:42:12.152583 31234 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:12.152179 31235 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe956765e5134132ae6b9be348e69271. Latest consensus state: current_term: 1 leader_uuid: "fe956765e5134132ae6b9be348e69271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe956765e5134132ae6b9be348e69271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37871 } } }
I20260926 16:42:12.152858 31235 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:12.155531 31240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:12.161001 31240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:12.172550 31240 catalog_manager.cc:1347] Generated new cluster ID: 42b9f270e5ce40708eb544ee44c0a804
I20260926 16:42:12.172686 31240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:12.211387 31240 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:12.213433 31240 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:12.223910 31240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe956765e5134132ae6b9be348e69271: Generated new TSK 0
I20260926 16:42:12.224836 31240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:12.233057 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31156
W20260926 16:42:12.234464 31218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32973
W20260926 16:42:12.235327 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31156: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:12.235541 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31156
W20260926 16:42:12.249869 31141 connection.cc:535] client connection to 127.0.0.1:37871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37871): Connection reset by peer (error 104)
2026-09-26T16:42:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:12 dist-test-slave-whjp krb5kdc[31254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:12 dist-test-slave-whjp krb5kdc[31254](info): set up 2 sockets
Sep 26 16:42:12 dist-test-slave-whjp krb5kdc[31254](info): commencing operation
krb5kdc: starting...
W20260926 16:42:13.273030 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:13 dist-test-slave-whjp krb5kdc[31254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:13.531307 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:13.872701 31267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:13.873004 31267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:13.873109 31267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:13.880944 31267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:13.881093 31267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:13.881140 31267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:13.881202 31267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:13.881247 31267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:13.892386 31267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:13.893775 31267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:13.895318 31267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:13.928485 31275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:13.928642 31273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:13.929586 31272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:13.929700 31267 server_base.cc:1034] running on GCE node
I20260926 16:42:13.930685 31267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:13.932309 31267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:13.933554 31267 hybrid_clock.cc:648] HybridClock initialized: now 1790440933933466 us; error 78 us; skew 500 ppm
I20260926 16:42:13.933946 31267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:13 dist-test-slave-whjp krb5kdc[31254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:13.937515 31267 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:13.939236 31267 webserver.cc:460] Webserver started at http://127.0.0.1:35455/ using document root <none> and password file <none>
I20260926 16:42:13.940126 31267 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:13.940284 31267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:13.940685 31267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:13.944293 31267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "b4f14fee1c3b47fab975cb73eb894da5"
format_stamp: "Formatted at 2026-09-26 16:42:13 on dist-test-slave-whjp"
server_key: "d08b24b7770d7fd2dc8812ef4c377250"
server_key_iv: "8a106232850e81b83133ca490dee7c5f"
server_key_version: "encryptionkey@0"
I20260926 16:42:13.945230 31267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "b4f14fee1c3b47fab975cb73eb894da5"
format_stamp: "Formatted at 2026-09-26 16:42:13 on dist-test-slave-whjp"
server_key: "d08b24b7770d7fd2dc8812ef4c377250"
server_key_iv: "8a106232850e81b83133ca490dee7c5f"
server_key_version: "encryptionkey@0"
I20260926 16:42:13.951154 31267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 16:42:13.955175 31282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:13.956648 31267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:13.956889 31267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "b4f14fee1c3b47fab975cb73eb894da5"
format_stamp: "Formatted at 2026-09-26 16:42:13 on dist-test-slave-whjp"
server_key: "d08b24b7770d7fd2dc8812ef4c377250"
server_key_iv: "8a106232850e81b83133ca490dee7c5f"
server_key_version: "encryptionkey@0"
I20260926 16:42:13.957108 31267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:13.963165 31267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:13.967691 31267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:13.968144 31267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:13.986752 31267 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44173
I20260926 16:42:13.986799 31333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44173 every 8 connection(s)
I20260926 16:42:13.988344 31267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:13.989622 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31267
I20260926 16:42:13.989807 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:13.990367 10680 external_mini_cluster.cc:1428] Setting key faa10e9d5d2755f8f6a238c5661d587a
I20260926 16:42:13.993525 31334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:13.998837 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:13.992289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57197 (local address 127.0.0.1:44173)
0926 16:42:13.993363 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:13.993376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:13.993422 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:42:13.995113 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:13.995138 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:13.995515 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:13.995913 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:13.997079 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:13.997086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:13.997101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:13.997128 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:13.997227 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:13.997251 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:13.997446 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:13.997560 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 16:42:13.997770 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":418,"threads_started":1}
I20260926 16:42:14.002782 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:14.006896 31334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5: Bootstrap starting.
I20260926 16:42:14.010393 31334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:14.010773 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:14.008116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56219 (local address 127.0.0.1:44173)
0926 16:42:14.008369 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:14.008377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:14.008854 (+ 477us) server_negotiation.cc:407] Connection header received
0926 16:42:14.009037 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:14.009040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:14.009094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:14.009178 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:14.010079 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:14.010085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:14.010090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:14.010108 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:14.010150 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:14.010165 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:14.010276 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:14.010402 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 16:42:14.010465 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:42:14.011595 31334 log.cc:824] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:14.014482 31334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5: No bootstrap required, opened a new log
I20260926 16:42:14.022341 31334 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f14fee1c3b47fab975cb73eb894da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44173 } }
I20260926 16:42:14.022814 31334 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:14.022912 31334 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f14fee1c3b47fab975cb73eb894da5, State: Initialized, Role: FOLLOWER
I20260926 16:42:14.023502 31334 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f14fee1c3b47fab975cb73eb894da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44173 } }
I20260926 16:42:14.023741 31334 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:14.023919 31334 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:14.024081 31334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:14.026314 31334 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f14fee1c3b47fab975cb73eb894da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44173 } }
I20260926 16:42:14.026877 31334 leader_election.cc:302] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f14fee1c3b47fab975cb73eb894da5; no voters:
I20260926 16:42:14.027523 31334 leader_election.cc:288] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:14.027657 31344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:14.028400 31344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [term 1 LEADER]: Becoming Leader. State: Replica: b4f14fee1c3b47fab975cb73eb894da5, State: Running, Role: LEADER
I20260926 16:42:14.028964 31344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f14fee1c3b47fab975cb73eb894da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44173 } }
I20260926 16:42:14.029719 31334 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:14.031714 31345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f14fee1c3b47fab975cb73eb894da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f14fee1c3b47fab975cb73eb894da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44173 } } }
I20260926 16:42:14.032162 31345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:14.032068 31346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f14fee1c3b47fab975cb73eb894da5. Latest consensus state: current_term: 1 leader_uuid: "b4f14fee1c3b47fab975cb73eb894da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f14fee1c3b47fab975cb73eb894da5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44173 } } }
I20260926 16:42:14.032367 31346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:14.034336 31350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:14.039873 31350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:14.048347 31350 catalog_manager.cc:1347] Generated new cluster ID: cd4e6999733b486cb37e7174bea12598
I20260926 16:42:14.048480 31350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:14.078919 31350 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:14.079461 31350 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:14.098644 31350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f14fee1c3b47fab975cb73eb894da5: Generated new TSK 0
I20260926 16:42:14.099660 31350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:14.108778 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31267
W20260926 16:42:14.109937 31329 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57197
W20260926 16:42:14.110623 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31267: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:14.110781 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31267
2026-09-26T16:42:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:14 dist-test-slave-whjp krb5kdc[31365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:14 dist-test-slave-whjp krb5kdc[31365](info): set up 2 sockets
Sep 26 16:42:14 dist-test-slave-whjp krb5kdc[31365](info): commencing operation
krb5kdc: starting...
W20260926 16:42:15.188839 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:15 dist-test-slave-whjp krb5kdc[31365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:15.436234 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:15.769110 31378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:15.769402 31378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:15.769531 31378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:15.776872 31378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:15.776997 31378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:15.777053 31378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:15.777138 31378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:15.777197 31378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:15.787282 31378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:15.788607 31378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:15.789873 31378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:15.801172 31383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:15.801213 31384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:15.801234 31386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:15.802094 31378 server_base.cc:1034] running on GCE node
I20260926 16:42:15.802860 31378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:15.804365 31378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:15.805567 31378 hybrid_clock.cc:648] HybridClock initialized: now 1790440935805522 us; error 41 us; skew 500 ppm
I20260926 16:42:15.805910 31378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:15 dist-test-slave-whjp krb5kdc[31365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:15.808898 31378 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:15.810338 31378 webserver.cc:460] Webserver started at http://127.0.0.1:42581/ using document root <none> and password file <none>
I20260926 16:42:15.810969 31378 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:15.811064 31378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:15.811312 31378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:15.814277 31378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "e1c9245d334f4e5f97316d7445aba2f0"
format_stamp: "Formatted at 2026-09-26 16:42:15 on dist-test-slave-whjp"
server_key: "68a5bb0b4dfb35e7faee2758d19aa80b"
server_key_iv: "66c99a55ebd7a81a8c05d880010d7018"
server_key_version: "encryptionkey@0"
I20260926 16:42:15.815002 31378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "e1c9245d334f4e5f97316d7445aba2f0"
format_stamp: "Formatted at 2026-09-26 16:42:15 on dist-test-slave-whjp"
server_key: "68a5bb0b4dfb35e7faee2758d19aa80b"
server_key_iv: "66c99a55ebd7a81a8c05d880010d7018"
server_key_version: "encryptionkey@0"
I20260926 16:42:15.820240 31378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:42:15.823684 31393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:15.825026 31378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:15.825212 31378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "e1c9245d334f4e5f97316d7445aba2f0"
format_stamp: "Formatted at 2026-09-26 16:42:15 on dist-test-slave-whjp"
server_key: "68a5bb0b4dfb35e7faee2758d19aa80b"
server_key_iv: "66c99a55ebd7a81a8c05d880010d7018"
server_key_version: "encryptionkey@0"
I20260926 16:42:15.825393 31378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:15.846835 31378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:15.851215 31378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:15.851478 31378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:15.870023 31378 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35567
I20260926 16:42:15.870023 31444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35567 every 8 connection(s)
I20260926 16:42:15.871448 31378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:15.875231 31445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:15.880548 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31378
I20260926 16:42:15.880731 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:15.881142 10680 external_mini_cluster.cc:1428] Setting key 428f912167d11fcdd0c40d72fbb08221
I20260926 16:42:15.887419 31445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0: Bootstrap starting.
I20260926 16:42:15.891368 31445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:15.892642 31445 log.cc:824] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:15.896353 31445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0: No bootstrap required, opened a new log
Sep 26 16:42:15 dist-test-slave-whjp krb5kdc[31365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:42:15.905051 31445 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c9245d334f4e5f97316d7445aba2f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260926 16:42:15.905471 31445 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:15.905591 31445 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c9245d334f4e5f97316d7445aba2f0, State: Initialized, Role: FOLLOWER
I20260926 16:42:15.905682 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.882979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59519 (local address 127.0.0.1:35567)
0926 16:42:15.883867 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.883883 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.883931 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:42:15.885736 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.885762 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.886187 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.886514 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:15.888502 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.889835 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.891720 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.892549 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.892711 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.897878 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:15.897886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:15.897902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:15.897946 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:15.901372 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:15.902173 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:15.902180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:15.902187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:15.902368 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:15.902746 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:15.902749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:15.902751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:15.903357 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:15.903673 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.904161 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.904598 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":390,"threads_started":1}
I20260926 16:42:15.906452 31445 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c9245d334f4e5f97316d7445aba2f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260926 16:42:15.906774 31445 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:15.906931 31445 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:15.907141 31445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:15.909588 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:15.910154 31445 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c9245d334f4e5f97316d7445aba2f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260926 16:42:15.910686 31445 leader_election.cc:302] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1c9245d334f4e5f97316d7445aba2f0; no voters:
I20260926 16:42:15.911278 31445 leader_election.cc:288] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:15.911422 31450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:15.913426 31445 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:15.920773 31450 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [term 1 LEADER]: Becoming Leader. State: Replica: e1c9245d334f4e5f97316d7445aba2f0, State: Running, Role: LEADER
I20260926 16:42:15.921608 31450 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c9245d334f4e5f97316d7445aba2f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } }
I20260926 16:42:15.924929 31451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1c9245d334f4e5f97316d7445aba2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c9245d334f4e5f97316d7445aba2f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } } }
I20260926 16:42:15.925323 31451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:15.925779 31452 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1c9245d334f4e5f97316d7445aba2f0. Latest consensus state: current_term: 1 leader_uuid: "e1c9245d334f4e5f97316d7445aba2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c9245d334f4e5f97316d7445aba2f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35567 } } }
I20260926 16:42:15.926061 31452 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0 [sys.catalog]: This master's current role is: LEADER
W20260926 16:42:15.932106 31471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:15.932250 31471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:15.932363 31469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:15.934211 31469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:15.938900 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:15.922806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40217 (local address 127.0.0.1:35567)
0926 16:42:15.923024 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:15.923031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:15.923280 (+ 249us) server_negotiation.cc:407] Connection header received
0926 16:42:15.923458 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:15.923462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:15.923531 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:15.923625 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:15.925284 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.926337 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.928072 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:15.928555 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:15.928649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:15.931588 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:15.931595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:15.931600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:15.931640 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:15.935630 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:15.936564 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:15.936570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:15.936574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:15.936744 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:15.937180 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:15.937185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:15.937188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:15.937671 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:15.937939 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:15.938296 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 16:42:15.938524 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 16:42:15.943320 31469 catalog_manager.cc:1347] Generated new cluster ID: a7940e4edd4a44df9ca05020286e367a
I20260926 16:42:15.943420 31469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:15.954581 31469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:15.956470 31469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:15.972491 31469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1c9245d334f4e5f97316d7445aba2f0: Generated new TSK 0
I20260926 16:42:15.973409 31469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:15.983752 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31378
I20260926 16:42:15.985011 31440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:16.077737 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31378
2026-09-26T16:42:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:16 dist-test-slave-whjp krb5kdc[31477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:16 dist-test-slave-whjp krb5kdc[31477](info): set up 2 sockets
Sep 26 16:42:16 dist-test-slave-whjp krb5kdc[31477](info): commencing operation
krb5kdc: starting...
W20260926 16:42:17.108860 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:17 dist-test-slave-whjp krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:17.359180 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:17.687817 31490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:17.688164 31490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:17.688282 31490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:17.695647 31490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:17.695760 31490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:17.695814 31490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:17.695916 31490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:17.695976 31490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:17.709362 31490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:17.710745 31490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:17.712109 31490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:17.723489 31498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:17.723603 31496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:17.723474 31495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:17.724875 31490 server_base.cc:1034] running on GCE node
I20260926 16:42:17.725754 31490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:17.727276 31490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:17.728485 31490 hybrid_clock.cc:648] HybridClock initialized: now 1790440937728424 us; error 51 us; skew 500 ppm
I20260926 16:42:17.728830 31490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:17 dist-test-slave-whjp krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:17.735996 31490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:17.737572 31490 webserver.cc:460] Webserver started at http://127.0.0.1:33443/ using document root <none> and password file <none>
I20260926 16:42:17.738276 31490 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:17.738395 31490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:17.738690 31490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:17.741870 31490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "0d64e41541934504949eb388dbaef514"
format_stamp: "Formatted at 2026-09-26 16:42:17 on dist-test-slave-whjp"
server_key: "b178907d23602e076058374b6883b2b3"
server_key_iv: "9839f70e372d280410cf42ad9ed19e14"
server_key_version: "encryptionkey@0"
I20260926 16:42:17.742671 31490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "0d64e41541934504949eb388dbaef514"
format_stamp: "Formatted at 2026-09-26 16:42:17 on dist-test-slave-whjp"
server_key: "b178907d23602e076058374b6883b2b3"
server_key_iv: "9839f70e372d280410cf42ad9ed19e14"
server_key_version: "encryptionkey@0"
I20260926 16:42:17.748061 31490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:42:17.751659 31505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:17.753129 31490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:17.753314 31490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "0d64e41541934504949eb388dbaef514"
format_stamp: "Formatted at 2026-09-26 16:42:17 on dist-test-slave-whjp"
server_key: "b178907d23602e076058374b6883b2b3"
server_key_iv: "9839f70e372d280410cf42ad9ed19e14"
server_key_version: "encryptionkey@0"
I20260926 16:42:17.753507 31490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:17.775234 31490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:17.779639 31490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:17.779973 31490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:17.798051 31490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44899
I20260926 16:42:17.798103 31556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44899 every 8 connection(s)
I20260926 16:42:17.799446 31490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:17.803387 31557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:17.804657 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31490
I20260926 16:42:17.804834 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:17.805392 10680 external_mini_cluster.cc:1428] Setting key 9b52ba57094a042d4a721d6142a99899
I20260926 16:42:17.817965 31557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514: Bootstrap starting.
I20260926 16:42:17.819669 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:17.807469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48967 (local address 127.0.0.1:44899)
0926 16:42:17.808345 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:17.808358 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:17.808407 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:42:17.810251 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:17.810284 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:17.810783 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:17.811157 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:17.811738 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.813170 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.815105 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.815895 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.816074 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:17.817339 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:17.817348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:17.817363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:17.817932 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:17.818023 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:17.818051 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:17.818153 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:17.818397 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 16:42:17.818655 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":413,"threads_started":1}
I20260926 16:42:17.821308 31557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:17.822443 31557 log.cc:824] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:17.823501 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:17.825449 31557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514: No bootstrap required, opened a new log
I20260926 16:42:17.834440 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:17.827812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43991 (local address 127.0.0.1:44899)
0926 16:42:17.827992 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:17.828000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:17.828360 (+ 360us) server_negotiation.cc:407] Connection header received
0926 16:42:17.828526 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:17.828530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:17.828653 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:17.828746 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:17.829372 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.830132 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.831938 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:17.832429 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:17.832594 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:17.833736 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:17.833741 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:17.833744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:17.833767 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:17.833827 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:17.833843 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:17.833921 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:17.834057 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 16:42:17.834138 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:42:17.834935 31557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64e41541934504949eb388dbaef514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260926 16:42:17.835420 31557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:17.835518 31557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d64e41541934504949eb388dbaef514, State: Initialized, Role: FOLLOWER
I20260926 16:42:17.836143 31557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64e41541934504949eb388dbaef514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260926 16:42:17.836374 31557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:17.836506 31557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:17.836673 31557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:17.838810 31557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64e41541934504949eb388dbaef514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260926 16:42:17.839315 31557 leader_election.cc:302] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d64e41541934504949eb388dbaef514; no voters:
I20260926 16:42:17.839887 31557 leader_election.cc:288] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:17.840013 31567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:17.840785 31567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [term 1 LEADER]: Becoming Leader. State: Replica: 0d64e41541934504949eb388dbaef514, State: Running, Role: LEADER
I20260926 16:42:17.841292 31567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64e41541934504949eb388dbaef514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } }
I20260926 16:42:17.842224 31557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:17.844017 31568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d64e41541934504949eb388dbaef514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64e41541934504949eb388dbaef514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } } }
I20260926 16:42:17.843992 31569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d64e41541934504949eb388dbaef514. Latest consensus state: current_term: 1 leader_uuid: "0d64e41541934504949eb388dbaef514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d64e41541934504949eb388dbaef514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44899 } } }
I20260926 16:42:17.844516 31569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:17.844506 31568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:17.846374 31572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:17.851294 31572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:17.860870 31572 catalog_manager.cc:1347] Generated new cluster ID: 8bc5f094eb1b449fae69c58ec920465b
I20260926 16:42:17.860982 31572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:17.872670 31572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:17.874096 31572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:17.880193 31572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d64e41541934504949eb388dbaef514: Generated new TSK 0
I20260926 16:42:17.881016 31572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:17.888824 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31490
W20260926 16:42:17.890965 31552 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48967
W20260926 16:42:17.891762 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31490: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:17.891923 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31490
2026-09-26T16:42:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1814 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:17 dist-test-slave-whjp krb5kdc[31588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:17 dist-test-slave-whjp krb5kdc[31588](info): set up 2 sockets
Sep 26 16:42:17 dist-test-slave-whjp krb5kdc[31588](info): commencing operation
krb5kdc: starting...
W20260926 16:42:18.925046 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:19 dist-test-slave-whjp krb5kdc[31588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:19.208297 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:19.568128 31601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:19.568446 31601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:19.568557 31601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:19.576880 31601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:19.577032 31601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:19.577080 31601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:19.577132 31601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:19.577181 31601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:19.587815 31601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:19.589181 31601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:19.590530 31601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:19.604100 31609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:19.604102 31607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:19.604239 31601 server_base.cc:1034] running on GCE node
W20260926 16:42:19.604100 31606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:19.605419 31601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:19.607103 31601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:19.608367 31601 hybrid_clock.cc:648] HybridClock initialized: now 1790440939608312 us; error 49 us; skew 500 ppm
I20260926 16:42:19.608821 31601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:19 dist-test-slave-whjp krb5kdc[31588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:19.612861 31601 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:19.615041 31601 webserver.cc:460] Webserver started at http://127.0.0.1:44157/ using document root <none> and password file <none>
I20260926 16:42:19.615878 31601 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:19.616022 31601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:19.616339 31601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:19.619690 31601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "03df983d67b74d77a47a1bf452708d46"
format_stamp: "Formatted at 2026-09-26 16:42:19 on dist-test-slave-whjp"
server_key: "b88defd2f3ab1b3dfa86af478b68fb37"
server_key_iv: "c4db258d2d59a750c07e24a23aab92be"
server_key_version: "encryptionkey@0"
I20260926 16:42:19.620944 31601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "03df983d67b74d77a47a1bf452708d46"
format_stamp: "Formatted at 2026-09-26 16:42:19 on dist-test-slave-whjp"
server_key: "b88defd2f3ab1b3dfa86af478b68fb37"
server_key_iv: "c4db258d2d59a750c07e24a23aab92be"
server_key_version: "encryptionkey@0"
I20260926 16:42:19.627364 31601 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:42:19.632097 31616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:19.633838 31601 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:42:19.634070 31601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "03df983d67b74d77a47a1bf452708d46"
format_stamp: "Formatted at 2026-09-26 16:42:19 on dist-test-slave-whjp"
server_key: "b88defd2f3ab1b3dfa86af478b68fb37"
server_key_iv: "c4db258d2d59a750c07e24a23aab92be"
server_key_version: "encryptionkey@0"
I20260926 16:42:19.634303 31601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:19.641867 31601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:19.646831 31601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:19.647177 31601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:19.668592 31601 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38557
I20260926 16:42:19.668641 31667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38557 every 8 connection(s)
I20260926 16:42:19.671170 31601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:19.674943 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31601
I20260926 16:42:19.675158 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:19.675673 10680 external_mini_cluster.cc:1428] Setting key 92a7c5f8d9813117d0ac856da142d11d
I20260926 16:42:19.676129 31668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:19.685509 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:19.678148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55625 (local address 127.0.0.1:38557)
0926 16:42:19.679495 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:19.679510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:19.679574 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:42:19.681659 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:19.681689 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:19.682128 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:19.682595 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:19.683721 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:19.683731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:19.683745 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:19.683768 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:19.683870 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:19.683906 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:19.684157 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:19.684292 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 16:42:19.684511 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":563,"threads_started":1}
I20260926 16:42:19.689512 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:19.690657 31668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46: Bootstrap starting.
I20260926 16:42:19.694067 31668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:19.695814 31668 log.cc:824] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:19.697244 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:19.693940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46737 (local address 127.0.0.1:38557)
0926 16:42:19.694158 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:19.694164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:19.694608 (+ 444us) server_negotiation.cc:407] Connection header received
0926 16:42:19.694766 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:19.694770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:19.694828 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:19.694921 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:19.696359 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:19.696365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:19.696376 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:19.696397 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:19.696442 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:19.696460 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:19.696578 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:19.696739 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 16:42:19.696813 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:42:19.699611 31668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46: No bootstrap required, opened a new log
I20260926 16:42:19.708595 31668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03df983d67b74d77a47a1bf452708d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38557 } }
I20260926 16:42:19.709041 31668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:19.709154 31668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03df983d67b74d77a47a1bf452708d46, State: Initialized, Role: FOLLOWER
I20260926 16:42:19.709879 31668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03df983d67b74d77a47a1bf452708d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38557 } }
I20260926 16:42:19.710201 31668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:19.710388 31668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:19.710556 31668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:19.712971 31668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03df983d67b74d77a47a1bf452708d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38557 } }
I20260926 16:42:19.713558 31668 leader_election.cc:302] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03df983d67b74d77a47a1bf452708d46; no voters:
I20260926 16:42:19.714182 31668 leader_election.cc:288] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:19.714385 31678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:19.715524 31678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [term 1 LEADER]: Becoming Leader. State: Replica: 03df983d67b74d77a47a1bf452708d46, State: Running, Role: LEADER
I20260926 16:42:19.716168 31678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03df983d67b74d77a47a1bf452708d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38557 } }
I20260926 16:42:19.716566 31668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:19.719118 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03df983d67b74d77a47a1bf452708d46. Latest consensus state: current_term: 1 leader_uuid: "03df983d67b74d77a47a1bf452708d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03df983d67b74d77a47a1bf452708d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38557 } } }
I20260926 16:42:19.719161 31679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03df983d67b74d77a47a1bf452708d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03df983d67b74d77a47a1bf452708d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38557 } } }
I20260926 16:42:19.719731 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:19.719787 31679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:19.724011 31684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:19.730003 31684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:19.740074 31684 catalog_manager.cc:1347] Generated new cluster ID: 217b3cd9798748119c17163392b1c5c0
I20260926 16:42:19.740236 31684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:19.766117 31684 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:19.766678 31684 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:19.781920 31684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03df983d67b74d77a47a1bf452708d46: Generated new TSK 0
I20260926 16:42:19.782917 31684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:19.799436 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31601
W20260926 16:42:19.800817 31663 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55625
W20260926 16:42:19.801564 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31601: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:19.801702 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31601
2026-09-26T16:42:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:19 dist-test-slave-whjp krb5kdc[31699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:19 dist-test-slave-whjp krb5kdc[31699](info): set up 2 sockets
Sep 26 16:42:19 dist-test-slave-whjp krb5kdc[31699](info): commencing operation
krb5kdc: starting...
W20260926 16:42:20.888906 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:21 dist-test-slave-whjp krb5kdc[31699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:21.147746 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:21.512387 31712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:21.512693 31712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:21.512826 31712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:21.520330 31712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:21.520462 31712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:21.520519 31712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:21.520605 31712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:21.520666 31712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:21.531569 31712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:21.533020 31712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:21.534318 31712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:21.547230 31717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:21.547288 31720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:21.547267 31718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:21.548313 31712 server_base.cc:1034] running on GCE node
I20260926 16:42:21.549273 31712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:21.550881 31712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:21.552125 31712 hybrid_clock.cc:648] HybridClock initialized: now 1790440941552073 us; error 38 us; skew 500 ppm
I20260926 16:42:21.552498 31712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:21 dist-test-slave-whjp krb5kdc[31699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:21.556056 31712 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:21.557696 31712 webserver.cc:460] Webserver started at http://127.0.0.1:38509/ using document root <none> and password file <none>
I20260926 16:42:21.558436 31712 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:21.558570 31712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:21.558866 31712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:21.562232 31712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "70241e2834ee4c179b6dac7e86ec7084"
format_stamp: "Formatted at 2026-09-26 16:42:21 on dist-test-slave-whjp"
server_key: "34e1dfee59e6d62a57edd508836b0a4a"
server_key_iv: "5ac575a9c43e6ae12620dc64c28f879d"
server_key_version: "encryptionkey@0"
I20260926 16:42:21.563068 31712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "70241e2834ee4c179b6dac7e86ec7084"
format_stamp: "Formatted at 2026-09-26 16:42:21 on dist-test-slave-whjp"
server_key: "34e1dfee59e6d62a57edd508836b0a4a"
server_key_iv: "5ac575a9c43e6ae12620dc64c28f879d"
server_key_version: "encryptionkey@0"
I20260926 16:42:21.568811 31712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 16:42:21.572737 31727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:21.574465 31712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:21.574681 31712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "70241e2834ee4c179b6dac7e86ec7084"
format_stamp: "Formatted at 2026-09-26 16:42:21 on dist-test-slave-whjp"
server_key: "34e1dfee59e6d62a57edd508836b0a4a"
server_key_iv: "5ac575a9c43e6ae12620dc64c28f879d"
server_key_version: "encryptionkey@0"
I20260926 16:42:21.574906 31712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:21.589725 31712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:21.595372 31712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:21.595817 31712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:21.616576 31712 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38235
I20260926 16:42:21.616652 31778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38235 every 8 connection(s)
I20260926 16:42:21.618093 31712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:21.622437 31779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:21.624034 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31712
I20260926 16:42:21.624233 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:21.624714 10680 external_mini_cluster.cc:1428] Setting key 1ecbf5c473ccfc007dc7ff22a9412060
I20260926 16:42:21.638967 31779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084: Bootstrap starting.
Sep 26 16:42:21 dist-test-slave-whjp krb5kdc[31699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:42:21.643492 31779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:21.644860 31779 log.cc:824] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:21.648514 31779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084: No bootstrap required, opened a new log
I20260926 16:42:21.653546 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:21.626836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:49205 (local address 127.0.0.1:38235)
0926 16:42:21.627881 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:21.627899 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:21.627956 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:42:21.629927 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:21.629960 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:21.630724 (+ 764us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:21.631152 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:21.633074 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.634628 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.636910 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.637745 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.637900 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:21.643482 (+ 5582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:21.643492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:21.643513 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:21.643564 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:21.647399 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:21.648539 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:21.648550 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:21.648559 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:21.648808 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:21.649479 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:21.649485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:21.649488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:21.650467 (+ 979us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:21.650833 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:21.651394 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 16:42:21.651877 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":471,"threads_started":1}
I20260926 16:42:21.658357 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:21.658807 31779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70241e2834ee4c179b6dac7e86ec7084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260926 16:42:21.659253 31779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:21.659358 31779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70241e2834ee4c179b6dac7e86ec7084, State: Initialized, Role: FOLLOWER
I20260926 16:42:21.660075 31779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70241e2834ee4c179b6dac7e86ec7084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260926 16:42:21.660327 31779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:21.660463 31779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:21.660619 31779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:21.663210 31779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70241e2834ee4c179b6dac7e86ec7084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260926 16:42:21.664057 31779 leader_election.cc:302] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70241e2834ee4c179b6dac7e86ec7084; no voters:
I20260926 16:42:21.664891 31779 leader_election.cc:288] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:21.665254 31789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:21.667528 31779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:21.668143 31789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [term 1 LEADER]: Becoming Leader. State: Replica: 70241e2834ee4c179b6dac7e86ec7084, State: Running, Role: LEADER
I20260926 16:42:21.668872 31789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70241e2834ee4c179b6dac7e86ec7084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260926 16:42:21.672616 31790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70241e2834ee4c179b6dac7e86ec7084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70241e2834ee4c179b6dac7e86ec7084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260926 16:42:21.673144 31790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:21.672049 31791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70241e2834ee4c179b6dac7e86ec7084. Latest consensus state: current_term: 1 leader_uuid: "70241e2834ee4c179b6dac7e86ec7084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70241e2834ee4c179b6dac7e86ec7084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260926 16:42:21.673923 31791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:21.684656 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:21.663217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:49573 (local address 127.0.0.1:38235)
0926 16:42:21.663437 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:21.663445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:21.663942 (+ 497us) server_negotiation.cc:407] Connection header received
0926 16:42:21.664209 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:21.664215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:21.664313 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:21.664409 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:21.666755 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.668146 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.670115 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:21.671009 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:21.671174 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:21.675101 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:21.675109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:21.675116 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:21.675171 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:21.680044 (+ 4873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:21.681723 (+ 1679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:21.681730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:21.681735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:21.681960 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:21.682641 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:21.682647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:21.682651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:21.683147 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:21.683489 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:21.683963 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 16:42:21.684210 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:42:21.688966 31797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:21.695302 31797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:21.704731 31797 catalog_manager.cc:1347] Generated new cluster ID: 36777214236e4dd0a541be8975a4bb05
I20260926 16:42:21.704869 31797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:21.726856 31797 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:21.729081 31797 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:21.737762 31797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70241e2834ee4c179b6dac7e86ec7084: Generated new TSK 0
I20260926 16:42:21.738826 31797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:21.755602 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31712
I20260926 16:42:21.757273 31774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:21.857668 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31712
W20260926 16:42:21.872598 31697 connection.cc:535] client connection to 127.0.0.1:38235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38235): Connection reset by peer (error 104)
2026-09-26T16:42:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:21 dist-test-slave-whjp krb5kdc[31812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:21 dist-test-slave-whjp krb5kdc[31812](info): set up 2 sockets
Sep 26 16:42:21 dist-test-slave-whjp krb5kdc[31812](info): commencing operation
krb5kdc: starting...
W20260926 16:42:22.912933 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:23 dist-test-slave-whjp krb5kdc[31812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:23.179035 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:23.534948 31825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:23.535261 31825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:23.535352 31825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:23.542778 31825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:23.542915 31825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:23.542961 31825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:23.543021 31825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:23.543064 31825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:23.553493 31825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:23.554814 31825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:23.556180 31825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:23.568459 31831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:23.569051 31830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:23.569131 31833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:23.570356 31825 server_base.cc:1034] running on GCE node
I20260926 16:42:23.572458 31825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:23.574582 31825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:23.575852 31825 hybrid_clock.cc:648] HybridClock initialized: now 1790440943575770 us; error 67 us; skew 500 ppm
I20260926 16:42:23.576256 31825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:23 dist-test-slave-whjp krb5kdc[31812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:23.585800 31825 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:23.587607 31825 webserver.cc:460] Webserver started at http://127.0.0.1:39071/ using document root <none> and password file <none>
I20260926 16:42:23.588480 31825 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:23.588605 31825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:23.588886 31825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:23.592235 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "fa1baa3dba3b4a769e22835491b260f1"
format_stamp: "Formatted at 2026-09-26 16:42:23 on dist-test-slave-whjp"
server_key: "1e94cd19a7ea32303803e422a30f0352"
server_key_iv: "44c0542f1ed4edb4ed04a22bb70145ab"
server_key_version: "encryptionkey@0"
I20260926 16:42:23.593025 31825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "fa1baa3dba3b4a769e22835491b260f1"
format_stamp: "Formatted at 2026-09-26 16:42:23 on dist-test-slave-whjp"
server_key: "1e94cd19a7ea32303803e422a30f0352"
server_key_iv: "44c0542f1ed4edb4ed04a22bb70145ab"
server_key_version: "encryptionkey@0"
I20260926 16:42:23.598775 31825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:23.602716 31840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:23.604409 31825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:23.604597 31825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "fa1baa3dba3b4a769e22835491b260f1"
format_stamp: "Formatted at 2026-09-26 16:42:23 on dist-test-slave-whjp"
server_key: "1e94cd19a7ea32303803e422a30f0352"
server_key_iv: "44c0542f1ed4edb4ed04a22bb70145ab"
server_key_version: "encryptionkey@0"
I20260926 16:42:23.604787 31825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:23.625062 31825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:23.629618 31825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:23.629876 31825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:23.647642 31825 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35553
I20260926 16:42:23.647645 31891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35553 every 8 connection(s)
I20260926 16:42:23.649166 31825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:23.653297 31892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:23.655505 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31825
I20260926 16:42:23.655674 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:23.656165 10680 external_mini_cluster.cc:1428] Setting key 34bee7338dc0181a1229ce0889252978
I20260926 16:42:23.666064 31892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1: Bootstrap starting.
I20260926 16:42:23.669570 31892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:23.670789 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:23.658093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:35871 (local address 127.0.0.1:35553)
0926 16:42:23.658997 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:23.659011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:23.659059 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:42:23.660914 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:23.660943 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:23.661564 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:23.662184 (+ 620us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:23.662915 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:23.664448 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:23.665881 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:23.666683 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:23.666873 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:23.668276 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:23.668284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:23.668300 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:23.668880 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:23.668968 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:23.668997 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:23.669146 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:23.669451 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 16:42:23.669693 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":384,"threads_started":1}
I20260926 16:42:23.670783 31892 log.cc:824] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:23.673954 31892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1: No bootstrap required, opened a new log
I20260926 16:42:23.675179 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:23.682953 31892 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1baa3dba3b4a769e22835491b260f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260926 16:42:23.683544 31892 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:23.683686 31892 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1baa3dba3b4a769e22835491b260f1, State: Initialized, Role: FOLLOWER
I20260926 16:42:23.684469 31892 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1baa3dba3b4a769e22835491b260f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260926 16:42:23.684757 31892 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:23.684906 31892 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:23.685091 31892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:23.687134 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:23.679859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:59881 (local address 127.0.0.1:35553)
0926 16:42:23.680169 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:23.680176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:23.680418 (+ 242us) server_negotiation.cc:407] Connection header received
0926 16:42:23.680679 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:23.680684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:23.680799 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:23.680899 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:23.681624 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:23.682404 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:23.684348 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:23.684899 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:23.685023 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:23.686325 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:23.686335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:23.686342 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:23.686364 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:23.686431 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:23.686449 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:23.686605 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:23.686721 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 16:42:23.686774 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 16:42:23.688275 31892 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1baa3dba3b4a769e22835491b260f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260926 16:42:23.688853 31892 leader_election.cc:302] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa1baa3dba3b4a769e22835491b260f1; no voters:
I20260926 16:42:23.689410 31892 leader_election.cc:288] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:23.689533 31902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:23.690155 31902 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [term 1 LEADER]: Becoming Leader. State: Replica: fa1baa3dba3b4a769e22835491b260f1, State: Running, Role: LEADER
I20260926 16:42:23.690697 31902 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1baa3dba3b4a769e22835491b260f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260926 16:42:23.691449 31892 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:23.693506 31903 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa1baa3dba3b4a769e22835491b260f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1baa3dba3b4a769e22835491b260f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } } }
I20260926 16:42:23.693766 31904 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa1baa3dba3b4a769e22835491b260f1. Latest consensus state: current_term: 1 leader_uuid: "fa1baa3dba3b4a769e22835491b260f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa1baa3dba3b4a769e22835491b260f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } } }
I20260926 16:42:23.694083 31904 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:23.694106 31903 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:23.694953 31908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:23.701226 31908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:23.713680 31908 catalog_manager.cc:1347] Generated new cluster ID: fcd03da57f4c429cb5a384dea63051de
I20260926 16:42:23.713822 31908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:23.728920 31908 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:23.730356 31908 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:23.737326 31908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa1baa3dba3b4a769e22835491b260f1: Generated new TSK 0
I20260926 16:42:23.738260 31908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:23.747623 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31825
W20260926 16:42:23.749415 31887 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:35871
W20260926 16:42:23.750332 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31825: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:23.750483 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31825
W20260926 16:42:23.764686 31810 connection.cc:535] client connection to 127.0.0.1:35553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35553): Connection reset by peer (error 104)
2026-09-26T16:42:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:23 dist-test-slave-whjp krb5kdc[31923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:23 dist-test-slave-whjp krb5kdc[31923](info): set up 2 sockets
Sep 26 16:42:23 dist-test-slave-whjp krb5kdc[31923](info): commencing operation
krb5kdc: starting...
W20260926 16:42:24.801020 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:24 dist-test-slave-whjp krb5kdc[31923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:25.064247 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:25.415146 31936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:25.415519 31936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:25.415666 31936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:25.423597 31936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:25.423776 31936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:25.423889 31936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:25.423987 31936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:25.424058 31936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:25.434712 31936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:25.436148 31936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:25.437592 31936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:25.450250 31941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:25.450273 31942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:25.450196 31944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:25.451481 31936 server_base.cc:1034] running on GCE node
I20260926 16:42:25.452410 31936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:25.453945 31936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:25.455207 31936 hybrid_clock.cc:648] HybridClock initialized: now 1790440945455143 us; error 42 us; skew 500 ppm
I20260926 16:42:25.455634 31936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:25 dist-test-slave-whjp krb5kdc[31923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:25.459333 31936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:25.461470 31936 webserver.cc:460] Webserver started at http://127.0.0.1:33417/ using document root <none> and password file <none>
I20260926 16:42:25.462252 31936 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:25.462400 31936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:25.462790 31936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:25.467077 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "30da08db991648f7b1c7416a8ea320b9"
format_stamp: "Formatted at 2026-09-26 16:42:25 on dist-test-slave-whjp"
server_key: "03dc76ed02abf5e8adf9214b4d189442"
server_key_iv: "939fb298985df58a8dc3a0fc318be159"
server_key_version: "encryptionkey@0"
I20260926 16:42:25.468163 31936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "30da08db991648f7b1c7416a8ea320b9"
format_stamp: "Formatted at 2026-09-26 16:42:25 on dist-test-slave-whjp"
server_key: "03dc76ed02abf5e8adf9214b4d189442"
server_key_iv: "939fb298985df58a8dc3a0fc318be159"
server_key_version: "encryptionkey@0"
I20260926 16:42:25.474323 31936 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:25.479276 31951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:25.481088 31936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 16:42:25.481312 31936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "30da08db991648f7b1c7416a8ea320b9"
format_stamp: "Formatted at 2026-09-26 16:42:25 on dist-test-slave-whjp"
server_key: "03dc76ed02abf5e8adf9214b4d189442"
server_key_iv: "939fb298985df58a8dc3a0fc318be159"
server_key_version: "encryptionkey@0"
I20260926 16:42:25.481559 31936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:25.488672 31936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:25.493659 31936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:25.494011 31936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:25.512754 31936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45863
I20260926 16:42:25.512775 32002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45863 every 8 connection(s)
I20260926 16:42:25.514904 31936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:25.519121 32003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:25.520884 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 31936
I20260926 16:42:25.521078 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:25.521652 10680 external_mini_cluster.cc:1428] Setting key 29f65cc72881dfc287d30b616732be68
I20260926 16:42:25.530511 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:25.523868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:54449 (local address 127.0.0.1:45863)
0926 16:42:25.524726 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:25.524739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:25.524786 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:42:25.526532 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:25.526567 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:25.527167 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:25.527709 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:25.528889 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:25.528896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:25.528912 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:25.528934 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:25.528997 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:25.529020 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:25.529205 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:25.529306 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 16:42:25.529522 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":354,"threads_started":1}
I20260926 16:42:25.534991 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:25.535374 32003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9: Bootstrap starting.
I20260926 16:42:25.539582 32003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:25.541006 32003 log.cc:824] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:25.544258 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:25.540637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:50055 (local address 127.0.0.1:45863)
0926 16:42:25.540848 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:25.540857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:25.541110 (+ 253us) server_negotiation.cc:407] Connection header received
0926 16:42:25.541332 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:25.541336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:25.541394 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:25.541495 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:25.542521 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:25.542525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:25.542529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:25.542545 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:25.542585 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:25.542601 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:25.542708 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:25.543757 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 16:42:25.543822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:42:25.545889 32003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9: No bootstrap required, opened a new log
I20260926 16:42:25.553855 32003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30da08db991648f7b1c7416a8ea320b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260926 16:42:25.554275 32003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:25.554370 32003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30da08db991648f7b1c7416a8ea320b9, State: Initialized, Role: FOLLOWER
I20260926 16:42:25.555061 32003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30da08db991648f7b1c7416a8ea320b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260926 16:42:25.555343 32003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:25.555496 32003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:25.555663 32003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:25.557961 32003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30da08db991648f7b1c7416a8ea320b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260926 16:42:25.558511 32003 leader_election.cc:302] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30da08db991648f7b1c7416a8ea320b9; no voters:
I20260926 16:42:25.559126 32003 leader_election.cc:288] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:25.559283 32013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:25.560000 32013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [term 1 LEADER]: Becoming Leader. State: Replica: 30da08db991648f7b1c7416a8ea320b9, State: Running, Role: LEADER
I20260926 16:42:25.560572 32013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30da08db991648f7b1c7416a8ea320b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260926 16:42:25.561297 32003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:25.563665 32015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30da08db991648f7b1c7416a8ea320b9. Latest consensus state: current_term: 1 leader_uuid: "30da08db991648f7b1c7416a8ea320b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30da08db991648f7b1c7416a8ea320b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } } }
I20260926 16:42:25.563655 32014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30da08db991648f7b1c7416a8ea320b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30da08db991648f7b1c7416a8ea320b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } } }
I20260926 16:42:25.564177 32014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:25.564177 32015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:25.565481 32022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:25.571501 32022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:25.581696 32022 catalog_manager.cc:1347] Generated new cluster ID: 249a71294dd64c45a6b5879b59bc38ec
I20260926 16:42:25.581830 32022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:25.621910 32022 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:25.622517 32022 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:25.636067 32022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30da08db991648f7b1c7416a8ea320b9: Generated new TSK 0
I20260926 16:42:25.637264 32022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:25.648155 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31936
W20260926 16:42:25.649346 31998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:54449
W20260926 16:42:25.650024 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 31936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:25.650177 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 31936
2026-09-26T16:42:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:25 dist-test-slave-whjp krb5kdc[32034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:25 dist-test-slave-whjp krb5kdc[32034](info): set up 2 sockets
Sep 26 16:42:25 dist-test-slave-whjp krb5kdc[32034](info): commencing operation
krb5kdc: starting...
W20260926 16:42:26.684938 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:26 dist-test-slave-whjp krb5kdc[32034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:26.956326 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:27.297900 32047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:27.298236 32047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:27.298374 32047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:27.306072 32047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:27.306226 32047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:27.306303 32047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:27.306382 32047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:27.306443 32047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:27.317296 32047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:27.318701 32047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:27.320137 32047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:27.332393 32052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:27.332419 32055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:27.332360 32053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:27.333783 32047 server_base.cc:1034] running on GCE node
I20260926 16:42:27.334664 32047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:27.336270 32047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:27.337563 32047 hybrid_clock.cc:648] HybridClock initialized: now 1790440947337468 us; error 46 us; skew 500 ppm
I20260926 16:42:27.337950 32047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:27 dist-test-slave-whjp krb5kdc[32034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:27.341413 32047 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:27.343083 32047 webserver.cc:460] Webserver started at http://127.0.0.1:41503/ using document root <none> and password file <none>
I20260926 16:42:27.343814 32047 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:27.343991 32047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:27.344309 32047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:27.347654 32047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "ce6adf894668429a96ce75b1f7778596"
format_stamp: "Formatted at 2026-09-26 16:42:27 on dist-test-slave-whjp"
server_key: "6f050a69d3b8ac0ae60924eeb939864c"
server_key_iv: "609c892084e8c870cd7f1a91ab4c2f5c"
server_key_version: "encryptionkey@0"
I20260926 16:42:27.348526 32047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "ce6adf894668429a96ce75b1f7778596"
format_stamp: "Formatted at 2026-09-26 16:42:27 on dist-test-slave-whjp"
server_key: "6f050a69d3b8ac0ae60924eeb939864c"
server_key_iv: "609c892084e8c870cd7f1a91ab4c2f5c"
server_key_version: "encryptionkey@0"
I20260926 16:42:27.354108 32047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:42:27.358573 32062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:27.360303 32047 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:42:27.360513 32047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "ce6adf894668429a96ce75b1f7778596"
format_stamp: "Formatted at 2026-09-26 16:42:27 on dist-test-slave-whjp"
server_key: "6f050a69d3b8ac0ae60924eeb939864c"
server_key_iv: "609c892084e8c870cd7f1a91ab4c2f5c"
server_key_version: "encryptionkey@0"
I20260926 16:42:27.360767 32047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:27.389873 32047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:27.394726 32047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:27.395057 32047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:27.413487 32047 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33057
I20260926 16:42:27.413509 32113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33057 every 8 connection(s)
I20260926 16:42:27.414888 32047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:27.418797 32114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:27.422030 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32047
I20260926 16:42:27.422205 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:27.422621 10680 external_mini_cluster.cc:1428] Setting key 452f2043f9928620cc230ec49313ac66
I20260926 16:42:27.431951 32114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596: Bootstrap starting.
I20260926 16:42:27.435849 32114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:27.437229 32114 log.cc:824] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:42:27 dist-test-slave-whjp krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:42:27.440725 32114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596: No bootstrap required, opened a new log
I20260926 16:42:27.448366 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:27.424770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:36719 (local address 127.0.0.1:33057)
0926 16:42:27.425537 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:27.425550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:27.425593 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:42:27.427311 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:27.427338 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:27.427887 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:27.428297 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:27.430494 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:27.432026 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:27.433931 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:27.434780 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:27.434957 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:27.439897 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:27.439908 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:27.439925 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:27.439987 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:27.443113 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:27.444278 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:27.444286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:27.444293 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:27.444553 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:27.445112 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:27.445116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:27.445118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:27.445816 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:27.446091 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:27.446733 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 16:42:27.447123 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":333,"threads_started":1}
I20260926 16:42:27.449733 32114 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6adf894668429a96ce75b1f7778596" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260926 16:42:27.450276 32114 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:27.450403 32114 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6adf894668429a96ce75b1f7778596, State: Initialized, Role: FOLLOWER
I20260926 16:42:27.451123 32114 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6adf894668429a96ce75b1f7778596" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260926 16:42:27.451435 32114 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:27.451586 32114 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:27.451725 32114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:27.453423 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:27.454438 32114 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6adf894668429a96ce75b1f7778596" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260926 16:42:27.454998 32114 leader_election.cc:302] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce6adf894668429a96ce75b1f7778596; no voters:
I20260926 16:42:27.455610 32114 leader_election.cc:288] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:27.455775 32119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:27.456786 32119 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [term 1 LEADER]: Becoming Leader. State: Replica: ce6adf894668429a96ce75b1f7778596, State: Running, Role: LEADER
I20260926 16:42:27.457659 32119 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6adf894668429a96ce75b1f7778596" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260926 16:42:27.459056 32114 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:27.461400 32122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce6adf894668429a96ce75b1f7778596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6adf894668429a96ce75b1f7778596" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260926 16:42:27.462013 32122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:27.462918 32121 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce6adf894668429a96ce75b1f7778596. Latest consensus state: current_term: 1 leader_uuid: "ce6adf894668429a96ce75b1f7778596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6adf894668429a96ce75b1f7778596" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260926 16:42:27.463475 32121 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:27.472328 32128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:27.479777 32128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:27.494988 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:27.468671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:38927 (local address 127.0.0.1:33057)
0926 16:42:27.468902 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:27.468909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:27.475670 (+ 6761us) server_negotiation.cc:407] Connection header received
0926 16:42:27.476059 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:27.476066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:27.476204 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:27.476298 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:27.478972 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:27.480395 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:27.481764 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:27.482231 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:27.482359 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:27.485419 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:27.485425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:27.485430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:27.485476 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:27.489369 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:27.490601 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:27.490607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:27.490611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:27.490789 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:27.491458 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:27.491464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:27.491468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:27.491989 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:27.494066 (+ 2077us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:27.494294 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 16:42:27.494543 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:42:27.501732 32128 catalog_manager.cc:1347] Generated new cluster ID: 76f7a108403f4f6eaa8c1b4dd630e92e
I20260926 16:42:27.501880 32128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:27.519119 32128 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:27.520656 32128 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:27.530023 32128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce6adf894668429a96ce75b1f7778596: Generated new TSK 0
I20260926 16:42:27.531195 32128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:27.542613 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32047
I20260926 16:42:27.543865 32109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:27.638569 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32047
W20260926 16:42:27.653894 32032 connection.cc:535] client connection to 127.0.0.1:33057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33057): Connection reset by peer (error 104)
2026-09-26T16:42:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:27 dist-test-slave-whjp krb5kdc[32147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:27 dist-test-slave-whjp krb5kdc[32147](info): set up 2 sockets
Sep 26 16:42:27 dist-test-slave-whjp krb5kdc[32147](info): commencing operation
krb5kdc: starting...
W20260926 16:42:28.680903 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:28 dist-test-slave-whjp krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:28.931401 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:29.261104 32160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:29.261440 32160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:29.261588 32160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:29.268983 32160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:29.269140 32160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:29.269227 32160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:29.269305 32160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:29.269364 32160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:29.279541 32160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:29.280939 32160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:29.282299 32160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:29.294173 32168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.294204 32166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:29.294390 32165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:29.295200 32160 server_base.cc:1034] running on GCE node
I20260926 16:42:29.296167 32160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:29.297755 32160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:29.299005 32160 hybrid_clock.cc:648] HybridClock initialized: now 1790440949298951 us; error 41 us; skew 500 ppm
I20260926 16:42:29.299384 32160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:29 dist-test-slave-whjp krb5kdc[32147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:29.307896 32160 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:29.309510 32160 webserver.cc:460] Webserver started at http://127.0.0.1:39239/ using document root <none> and password file <none>
I20260926 16:42:29.310242 32160 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:29.310357 32160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:29.310830 32160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:29.314612 32160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "b086962f1447458e86d283d12f7afa5c"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-whjp"
server_key: "3a9504c719a18b764b8544532893bf96"
server_key_iv: "2e941da7d792cba3a3d84252b01c6322"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.315492 32160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "b086962f1447458e86d283d12f7afa5c"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-whjp"
server_key: "3a9504c719a18b764b8544532893bf96"
server_key_iv: "2e941da7d792cba3a3d84252b01c6322"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.321305 32160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:42:29.325064 32175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:29.326654 32160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:29.326828 32160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "b086962f1447458e86d283d12f7afa5c"
format_stamp: "Formatted at 2026-09-26 16:42:29 on dist-test-slave-whjp"
server_key: "3a9504c719a18b764b8544532893bf96"
server_key_iv: "2e941da7d792cba3a3d84252b01c6322"
server_key_version: "encryptionkey@0"
I20260926 16:42:29.327023 32160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:29.376132 32160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:29.381997 32160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:29.382292 32160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:29.405475 32160 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45881
I20260926 16:42:29.405501 32226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45881 every 8 connection(s)
I20260926 16:42:29.406950 32160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:29.410899 32227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:29.417243 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32160
I20260926 16:42:29.417436 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:29.417869 10680 external_mini_cluster.cc:1428] Setting key 10bf2eed338ba15c61af6e7902b995bc
I20260926 16:42:29.428550 32227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c: Bootstrap starting.
I20260926 16:42:29.433866 32227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:29.434582 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.420223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:32841 (local address 127.0.0.1:45881)
0926 16:42:29.421214 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.421228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.421277 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:42:29.423087 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.423115 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.423591 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.424029 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.424732 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.426209 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.428172 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.428975 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.429857 (+ 882us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.432015 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.432029 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.432051 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:29.432666 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.432754 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:29.432789 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.432953 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.433268 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.433452 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":437,"threads_started":1}
I20260926 16:42:29.435568 32227 log.cc:824] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:29.441928 32227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c: No bootstrap required, opened a new log
I20260926 16:42:29.442060 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:29.456709 32227 raft_consensus.cc:348] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b086962f1447458e86d283d12f7afa5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45881 } }
I20260926 16:42:29.458135 32227 raft_consensus.cc:374] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:29.458308 32227 raft_consensus.cc:729] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b086962f1447458e86d283d12f7afa5c, State: Initialized, Role: FOLLOWER
I20260926 16:42:29.459666 32227 consensus_queue.cc:260] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b086962f1447458e86d283d12f7afa5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45881 } }
I20260926 16:42:29.460062 32227 raft_consensus.cc:388] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:29.460242 32227 raft_consensus.cc:482] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:29.460464 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:29.448258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:35553 (local address 127.0.0.1:45881)
0926 16:42:29.448497 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:29.448504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:29.448906 (+ 402us) server_negotiation.cc:407] Connection header received
0926 16:42:29.449123 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:29.449127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:29.449544 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:29.449860 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:29.452878 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.454169 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.456355 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:29.457559 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:29.457835 (+ 276us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:29.459462 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:29.459469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:29.459473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:29.459501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:29.459579 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:29.459607 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:29.459751 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:29.460035 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 16:42:29.460104 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:42:29.460466 32227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:29.463040 32227 raft_consensus.cc:504] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b086962f1447458e86d283d12f7afa5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45881 } }
I20260926 16:42:29.463613 32227 leader_election.cc:302] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b086962f1447458e86d283d12f7afa5c; no voters:
I20260926 16:42:29.464305 32227 leader_election.cc:288] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:29.464497 32237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:29.465287 32237 raft_consensus.cc:686] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [term 1 LEADER]: Becoming Leader. State: Replica: b086962f1447458e86d283d12f7afa5c, State: Running, Role: LEADER
I20260926 16:42:29.466449 32227 sys_catalog.cc:564] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:29.466372 32237 consensus_queue.cc:237] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b086962f1447458e86d283d12f7afa5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45881 } }
I20260926 16:42:29.470969 32238 sys_catalog.cc:455] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b086962f1447458e86d283d12f7afa5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b086962f1447458e86d283d12f7afa5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45881 } } }
I20260926 16:42:29.471514 32238 sys_catalog.cc:458] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:29.471220 32239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b086962f1447458e86d283d12f7afa5c. Latest consensus state: current_term: 1 leader_uuid: "b086962f1447458e86d283d12f7afa5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b086962f1447458e86d283d12f7afa5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45881 } } }
I20260926 16:42:29.471671 32239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:29.474836 32246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:29.481508 32246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:29.492017 32246 catalog_manager.cc:1347] Generated new cluster ID: 841ad4f4b15b4fc3839eaa3d1adab27c
I20260926 16:42:29.492154 32246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:29.516820 32246 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:29.518817 32246 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:29.531912 32246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b086962f1447458e86d283d12f7afa5c: Generated new TSK 0
I20260926 16:42:29.533468 32246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:29.551424 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32160
W20260926 16:42:29.553223 32222 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:32841
W20260926 16:42:29.553963 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32160: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:29.554127 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32160
W20260926 16:42:29.570137 32145 connection.cc:535] client connection to 127.0.0.1:45881 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45881): Connection reset by peer (error 104)
2026-09-26T16:42:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:29 dist-test-slave-whjp krb5kdc[32258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:29 dist-test-slave-whjp krb5kdc[32258](info): set up 2 sockets
Sep 26 16:42:29 dist-test-slave-whjp krb5kdc[32258](info): commencing operation
krb5kdc: starting...
W20260926 16:42:30.588860 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:30 dist-test-slave-whjp krb5kdc[32258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:42:30.834890 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:31.164803 32271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:31.165117 32271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:31.165238 32271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:31.172679 32271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:31.172813 32271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:31.172873 32271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:31.172960 32271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:31.173022 32271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:31.183508 32271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:31.184870 32271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:31.186192 32271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:31.197898 32276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.198019 32277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:31.198303 32279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:31.198789 32271 server_base.cc:1034] running on GCE node
I20260926 16:42:31.199580 32271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:31.201069 32271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:31.202270 32271 hybrid_clock.cc:648] HybridClock initialized: now 1790440951202223 us; error 40 us; skew 500 ppm
I20260926 16:42:31.202613 32271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:31 dist-test-slave-whjp krb5kdc[32258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:31.205852 32271 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:42:31.207487 32271 webserver.cc:460] Webserver started at http://127.0.0.1:40453/ using document root <none> and password file <none>
I20260926 16:42:31.208222 32271 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:31.208334 32271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:31.208616 32271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:31.211856 32271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "ff67d398ff9a444fbc1edde4a5342ee1"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-whjp"
server_key: "635c53db56b569b0c82d8c5464344866"
server_key_iv: "bb2c4339d7143b1781606f32976bc128"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.212729 32271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "ff67d398ff9a444fbc1edde4a5342ee1"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-whjp"
server_key: "635c53db56b569b0c82d8c5464344866"
server_key_iv: "bb2c4339d7143b1781606f32976bc128"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.218487 32271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:42:31.222527 32286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:31.224216 32271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:31.224401 32271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "ff67d398ff9a444fbc1edde4a5342ee1"
format_stamp: "Formatted at 2026-09-26 16:42:31 on dist-test-slave-whjp"
server_key: "635c53db56b569b0c82d8c5464344866"
server_key_iv: "bb2c4339d7143b1781606f32976bc128"
server_key_version: "encryptionkey@0"
I20260926 16:42:31.224617 32271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:31.230960 32271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:31.235245 32271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:31.235536 32271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:31.253816 32271 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38173
I20260926 16:42:31.253839 32337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38173 every 8 connection(s)
I20260926 16:42:31.255286 32271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:31.258462 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32271
I20260926 16:42:31.258637 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:31.259111 10680 external_mini_cluster.cc:1428] Setting key 497679f17c9f439ae207a67e4e1e624c
I20260926 16:42:31.259384 32338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:31.267776 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.260852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:56057 (local address 127.0.0.1:38173)
0926 16:42:31.261624 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.261643 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.261685 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:42:31.264121 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.264153 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.264578 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.264985 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:31.266106 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:31.266120 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:31.266141 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:31.266159 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:31.266235 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:31.266258 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:31.266451 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.266562 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.266809 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":322,"threads_started":1}
I20260926 16:42:31.271476 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:31.273447 32338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1: Bootstrap starting.
I20260926 16:42:31.277596 32338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:31.279026 32338 log.cc:824] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:31.279223 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:31.276752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.129:43775 (local address 127.0.0.1:38173)
0926 16:42:31.276908 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:31.276914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:31.277287 (+ 373us) server_negotiation.cc:407] Connection header received
0926 16:42:31.277473 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:31.277476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:31.277532 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:31.277611 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:31.278510 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:31.278514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:31.278517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:31.278533 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:31.278574 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:31.278589 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:31.278717 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:31.278830 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 16:42:31.278883 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 16:42:31.282033 32338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1: No bootstrap required, opened a new log
I20260926 16:42:31.290037 32338 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260926 16:42:31.290482 32338 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:31.290591 32338 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff67d398ff9a444fbc1edde4a5342ee1, State: Initialized, Role: FOLLOWER
I20260926 16:42:31.291203 32338 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260926 16:42:31.291414 32338 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:31.291549 32338 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:31.291704 32338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:31.294247 32338 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260926 16:42:31.294955 32338 leader_election.cc:302] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff67d398ff9a444fbc1edde4a5342ee1; no voters:
I20260926 16:42:31.295872 32338 leader_election.cc:288] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:31.296029 32348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:31.296702 32348 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [term 1 LEADER]: Becoming Leader. State: Replica: ff67d398ff9a444fbc1edde4a5342ee1, State: Running, Role: LEADER
I20260926 16:42:31.297340 32348 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } }
I20260926 16:42:31.298049 32338 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:31.300062 32349 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } } }
I20260926 16:42:31.300563 32349 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:31.301781 32357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:31.302289 32350 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff67d398ff9a444fbc1edde4a5342ee1. Latest consensus state: current_term: 1 leader_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff67d398ff9a444fbc1edde4a5342ee1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38173 } } }
I20260926 16:42:31.302726 32350 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:31.307893 32357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:31.316900 32357 catalog_manager.cc:1347] Generated new cluster ID: 9eaffbe9f0f14882b51546412e557c61
I20260926 16:42:31.317063 32357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:31.330281 32357 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:31.330749 32357 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:31.340766 32357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff67d398ff9a444fbc1edde4a5342ee1: Generated new TSK 0
I20260926 16:42:31.341679 32357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:31.349047 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32271
W20260926 16:42:31.350108 32333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.129:56057
W20260926 16:42:31.350804 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32271: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:31.350958 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32271
2026-09-26T16:42:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1790 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:31 dist-test-slave-whjp krb5kdc[32369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:31 dist-test-slave-whjp krb5kdc[32369](info): set up 2 sockets
Sep 26 16:42:31 dist-test-slave-whjp krb5kdc[32369](info): commencing operation
krb5kdc: starting...
W20260926 16:42:32.388895 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:32 dist-test-slave-whjp krb5kdc[32369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:32.648331 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40715
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:43205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:40715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:32.982816 32382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:32.983105 32382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:32.983238 32382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:32.990661 32382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:32.990792 32382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:32.990849 32382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:32.990936 32382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:32.990993 32382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:33.001137 32382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:43205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:40715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:40715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:33.002434 32382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:33.003780 32382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:33.015923 32387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.015864 32388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:33.015893 32390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:33.017230 32382 server_base.cc:1034] running on GCE node
I20260926 16:42:33.018023 32382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:33.019548 32382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:33.020802 32382 hybrid_clock.cc:648] HybridClock initialized: now 1790440953020755 us; error 31 us; skew 500 ppm
I20260926 16:42:33.021196 32382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:33 dist-test-slave-whjp krb5kdc[32369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440953, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:33.024246 32382 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:33.026301 32382 webserver.cc:460] Webserver started at http://127.10.110.62:42209/ using document root <none> and password file <none>
I20260926 16:42:33.026981 32382 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:33.027102 32382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:33.027396 32382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:33.030408 32382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "2186ab4c3d954f049e655a965f08a968"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-whjp"
server_key: "a12f34806da2ebdcef73f80954f6d5ea"
server_key_iv: "f9c7f662a6769fdcff222a0846f6609e"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.031137 32382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "2186ab4c3d954f049e655a965f08a968"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-whjp"
server_key: "a12f34806da2ebdcef73f80954f6d5ea"
server_key_iv: "f9c7f662a6769fdcff222a0846f6609e"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.036495 32382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:42:33.040596 32397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:33.041983 32382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:33.042150 32382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "2186ab4c3d954f049e655a965f08a968"
format_stamp: "Formatted at 2026-09-26 16:42:33 on dist-test-slave-whjp"
server_key: "a12f34806da2ebdcef73f80954f6d5ea"
server_key_iv: "f9c7f662a6769fdcff222a0846f6609e"
server_key_version: "encryptionkey@0"
I20260926 16:42:33.042343 32382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:33.063714 32382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:33.068017 32382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:33.068301 32382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:33.085708 32382 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:40715
I20260926 16:42:33.085712 32448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:40715 every 8 connection(s)
I20260926 16:42:33.087134 32382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:33.090950 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:33.092736 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32382
I20260926 16:42:33.092924 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:33.093485 10680 external_mini_cluster.cc:1428] Setting key 8b051eaa4788c1f6c559d2237edcffc0
I20260926 16:42:33.125527 32449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968: Bootstrap starting.
Sep 26 16:42:33 dist-test-slave-whjp krb5kdc[32369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:33.129621 32449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:33.130944 32449 log.cc:824] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:33.134933 32449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968: No bootstrap required, opened a new log
I20260926 16:42:33.139103 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.101051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53757 (local address 127.10.110.62:40715)
0926 16:42:33.102708 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.102724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.102780 (+ 56us) server_negotiation.cc:407] Connection header received
0926 16:42:33.105167 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.105201 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.105770 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.106175 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:33.108439 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.118305 (+ 9866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.121890 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.122873 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.123040 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.128428 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:33.128438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:33.128456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:33.128507 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:33.132465 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:33.133856 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:33.133867 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:33.133877 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:33.134154 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:33.134814 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:33.134820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:33.134824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:33.135703 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:33.136120 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:33.136657 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 16:42:33.137055 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":731,"threads_started":1}
I20260926 16:42:33.144608 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:33.145922 32449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2186ab4c3d954f049e655a965f08a968" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40715 } }
I20260926 16:42:33.146356 32449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:33.146456 32449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2186ab4c3d954f049e655a965f08a968, State: Initialized, Role: FOLLOWER
I20260926 16:42:33.147110 32449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2186ab4c3d954f049e655a965f08a968" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40715 } }
I20260926 16:42:33.147367 32449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:33.147498 32449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:33.147655 32449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:33.150264 32449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2186ab4c3d954f049e655a965f08a968" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40715 } }
I20260926 16:42:33.150856 32449 leader_election.cc:302] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2186ab4c3d954f049e655a965f08a968; no voters:
I20260926 16:42:33.151481 32449 leader_election.cc:288] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:33.151604 32459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:33.154940 32449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:33.158820 32459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [term 1 LEADER]: Becoming Leader. State: Replica: 2186ab4c3d954f049e655a965f08a968, State: Running, Role: LEADER
I20260926 16:42:33.159574 32459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2186ab4c3d954f049e655a965f08a968" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40715 } }
I20260926 16:42:33.165968 32460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2186ab4c3d954f049e655a965f08a968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2186ab4c3d954f049e655a965f08a968" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40715 } } }
I20260926 16:42:33.166492 32460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.166903 32461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2186ab4c3d954f049e655a965f08a968. Latest consensus state: current_term: 1 leader_uuid: "2186ab4c3d954f049e655a965f08a968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2186ab4c3d954f049e655a965f08a968" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 40715 } } }
I20260926 16:42:33.167222 32461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:33.169421 32474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:33.172528 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:33.149733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37117 (local address 127.10.110.62:40715)
0926 16:42:33.149919 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:33.149925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:33.156564 (+ 6639us) server_negotiation.cc:407] Connection header received
0926 16:42:33.156981 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:33.156987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:33.157108 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:33.157199 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:33.159170 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.160070 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.161349 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:33.161808 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:33.161923 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:33.164682 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:33.164689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:33.164694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:33.164737 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:33.168797 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:33.169951 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:33.169957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:33.169960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:33.170109 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:33.170613 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:33.170618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:33.170621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:33.171033 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:33.171321 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:33.171720 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 16:42:33.171948 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:42:33.174925 32474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:33.184242 32474 catalog_manager.cc:1347] Generated new cluster ID: f9912c96de7a4357b65a64af355e219c
I20260926 16:42:33.184370 32474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:33.199517 32474 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:33.201498 32474 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:33.216230 32474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2186ab4c3d954f049e655a965f08a968: Generated new TSK 0
I20260926 16:42:33.217470 32474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:33.231348 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32382
I20260926 16:42:33.232513 32444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:33.325165 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32382
W20260926 16:42:33.339462 32367 connection.cc:535] client connection to 127.10.110.62:40715 recv error: Network error: failed to read from TLS socket (remote: 127.10.110.62:40715): Connection reset by peer (error 104)
2026-09-26T16:42:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:33 dist-test-slave-whjp krb5kdc[32481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:33 dist-test-slave-whjp krb5kdc[32481](info): set up 2 sockets
Sep 26 16:42:33 dist-test-slave-whjp krb5kdc[32481](info): commencing operation
krb5kdc: starting...
W20260926 16:42:34.374055 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:34 dist-test-slave-whjp krb5kdc[32481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:34.620633 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:35007
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:34.951359 32494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:34.951658 32494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:34.951750 32494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:34.959038 32494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:34.959160 32494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:34.959213 32494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:34.959272 32494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:34.959316 32494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:34.969771 32494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:35007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:34.971469 32494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:34.973040 32494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:34.985149 32500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:34.985193 32499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:34.985280 32502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:34.985982 32494 server_base.cc:1034] running on GCE node
I20260926 16:42:34.986804 32494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:34.988312 32494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:34.989521 32494 hybrid_clock.cc:648] HybridClock initialized: now 1790440954989451 us; error 58 us; skew 500 ppm
I20260926 16:42:34.989866 32494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:34 dist-test-slave-whjp krb5kdc[32481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440954, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:34.995875 32494 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:34.997591 32494 webserver.cc:460] Webserver started at http://127.10.110.62:44061/ using document root <none> and password file <none>
I20260926 16:42:34.998315 32494 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:34.998440 32494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:34.998740 32494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:35.001858 32494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "d571e2555d9149be8879270ca68f76f9"
format_stamp: "Formatted at 2026-09-26 16:42:34 on dist-test-slave-whjp"
server_key: "1bcf4b776ec01924ccd180feebb94bbe"
server_key_iv: "f3afbb4cfde732a1556256389eca1139"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.002651 32494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "d571e2555d9149be8879270ca68f76f9"
format_stamp: "Formatted at 2026-09-26 16:42:34 on dist-test-slave-whjp"
server_key: "1bcf4b776ec01924ccd180feebb94bbe"
server_key_iv: "f3afbb4cfde732a1556256389eca1139"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.008258 32494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:42:35.012034 32509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:35.013505 32494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:35.013687 32494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "d571e2555d9149be8879270ca68f76f9"
format_stamp: "Formatted at 2026-09-26 16:42:34 on dist-test-slave-whjp"
server_key: "1bcf4b776ec01924ccd180feebb94bbe"
server_key_iv: "f3afbb4cfde732a1556256389eca1139"
server_key_version: "encryptionkey@0"
I20260926 16:42:35.013880 32494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:35.034303 32494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:35.038712 32494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:35.038985 32494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:35.056052 32494 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:35007
I20260926 16:42:35.056061 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:35007 every 8 connection(s)
I20260926 16:42:35.057425 32494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:35.061476 32561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:35.064561 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32494
I20260926 16:42:35.064749 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:35.065269 10680 external_mini_cluster.cc:1428] Setting key 31e5615d44ea330ee6fbaad4c1936194
I20260926 16:42:35.074012 32561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9: Bootstrap starting.
I20260926 16:42:35.077600 32561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:35.078666 32561 log.cc:824] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:35.078809 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:35.067219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33857 (local address 127.10.110.62:35007)
0926 16:42:35.068078 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:35.068091 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:35.068134 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:42:35.069810 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:35.069837 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:35.070263 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:35.070592 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:35.071177 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.072574 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.074356 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.075043 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.075169 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:35.076454 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:35.076462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:35.076476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:35.076998 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:35.077076 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:35.077101 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:35.077212 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:35.077605 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 16:42:35.077801 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":402,"threads_started":1}
I20260926 16:42:35.081625 32561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9: No bootstrap required, opened a new log
I20260926 16:42:35.082850 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:35.089797 32561 raft_consensus.cc:348] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571e2555d9149be8879270ca68f76f9" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35007 } }
I20260926 16:42:35.090188 32561 raft_consensus.cc:374] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:35.090307 32561 raft_consensus.cc:729] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d571e2555d9149be8879270ca68f76f9, State: Initialized, Role: FOLLOWER
I20260926 16:42:35.090931 32561 consensus_queue.cc:260] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571e2555d9149be8879270ca68f76f9" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35007 } }
I20260926 16:42:35.091233 32561 raft_consensus.cc:388] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:35.091379 32561 raft_consensus.cc:482] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:35.091562 32561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:35.093307 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:35.087122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39949 (local address 127.10.110.62:35007)
0926 16:42:35.087347 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:35.087353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:35.087717 (+ 364us) server_negotiation.cc:407] Connection header received
0926 16:42:35.087906 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:35.087910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:35.087978 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:35.088073 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:35.088564 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.089321 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.090717 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:35.091354 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:35.091486 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:35.092594 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:35.092599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:35.092602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:35.092619 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:35.092683 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:35.092700 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:35.092802 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:35.092987 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 16:42:35.093046 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:42:35.094465 32561 raft_consensus.cc:504] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571e2555d9149be8879270ca68f76f9" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35007 } }
I20260926 16:42:35.095002 32561 leader_election.cc:302] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d571e2555d9149be8879270ca68f76f9; no voters:
I20260926 16:42:35.095610 32561 leader_election.cc:288] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:35.095777 32571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:35.096508 32571 raft_consensus.cc:686] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [term 1 LEADER]: Becoming Leader. State: Replica: d571e2555d9149be8879270ca68f76f9, State: Running, Role: LEADER
I20260926 16:42:35.097033 32571 consensus_queue.cc:237] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571e2555d9149be8879270ca68f76f9" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35007 } }
I20260926 16:42:35.097723 32561 sys_catalog.cc:564] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:35.099771 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d571e2555d9149be8879270ca68f76f9. Latest consensus state: current_term: 1 leader_uuid: "d571e2555d9149be8879270ca68f76f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571e2555d9149be8879270ca68f76f9" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35007 } } }
I20260926 16:42:35.100241 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:35.099776 32572 sys_catalog.cc:455] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d571e2555d9149be8879270ca68f76f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d571e2555d9149be8879270ca68f76f9" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35007 } } }
I20260926 16:42:35.101564 32572 sys_catalog.cc:458] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:35.101825 32576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:35.106611 32576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:35.116907 32576 catalog_manager.cc:1347] Generated new cluster ID: 1c634119e81d49e3a792256571bcd8f4
I20260926 16:42:35.117038 32576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:35.154762 32576 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:35.156682 32576 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:35.169744 32576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d571e2555d9149be8879270ca68f76f9: Generated new TSK 0
I20260926 16:42:35.170696 32576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:35.184926 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32494
W20260926 16:42:35.186136 32556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33857
W20260926 16:42:35.186883 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:35.187021 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32494
W20260926 16:42:35.200834 32479 connection.cc:535] client connection to 127.10.110.62:35007 recv error: Network error: failed to read from TLS socket (remote: 127.10.110.62:35007): Connection reset by peer (error 104)
2026-09-26T16:42:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:35 dist-test-slave-whjp krb5kdc[32592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:35 dist-test-slave-whjp krb5kdc[32592](info): set up 2 sockets
Sep 26 16:42:35 dist-test-slave-whjp krb5kdc[32592](info): commencing operation
krb5kdc: starting...
W20260926 16:42:36.209364 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:36 dist-test-slave-whjp krb5kdc[32592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:36.458541 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38165
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:38165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:36.787688 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:36.788017 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:36.788292 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:36.796029 32605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:36.796149 32605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:36.796339 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:36.796437 32605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:36.796496 32605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:36.806677 32605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:38165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:38165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:36.808053 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:36.809350 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:36.820979 32610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:36.821007 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:36.821079 32613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:36.821925 32605 server_base.cc:1034] running on GCE node
I20260926 16:42:36.822693 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:36.824103 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:36.825309 32605 hybrid_clock.cc:648] HybridClock initialized: now 1790440956825268 us; error 37 us; skew 500 ppm
I20260926 16:42:36.825621 32605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:36 dist-test-slave-whjp krb5kdc[32592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440956, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:36.828553 32605 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:36.830036 32605 webserver.cc:460] Webserver started at http://127.10.110.62:34299/ using document root <none> and password file <none>
I20260926 16:42:36.830713 32605 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:36.830825 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:36.831097 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:36.834137 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "7638a2fbd40b4b67b103ab356ab635ae"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-whjp"
server_key: "4befa5e127293dec6c0f5fcca829638a"
server_key_iv: "cd6de4b4f59fa87026b6cf48ef6b4add"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.834882 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "7638a2fbd40b4b67b103ab356ab635ae"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-whjp"
server_key: "4befa5e127293dec6c0f5fcca829638a"
server_key_iv: "cd6de4b4f59fa87026b6cf48ef6b4add"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.840104 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 16:42:36.843703 32620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:36.845232 32605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:36.845412 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "7638a2fbd40b4b67b103ab356ab635ae"
format_stamp: "Formatted at 2026-09-26 16:42:36 on dist-test-slave-whjp"
server_key: "4befa5e127293dec6c0f5fcca829638a"
server_key_iv: "cd6de4b4f59fa87026b6cf48ef6b4add"
server_key_version: "encryptionkey@0"
I20260926 16:42:36.845623 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:36.851904 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:36.856146 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:36.856424 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:36.873883 32605 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:38165
I20260926 16:42:36.873904 32671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:38165 every 8 connection(s)
I20260926 16:42:36.875273 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:36.878907 32672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:36.882550 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32605
I20260926 16:42:36.882728 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:36.883285 10680 external_mini_cluster.cc:1428] Setting key 61c58fcb0d0317c6462575e6820349a0
I20260926 16:42:36.891137 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.885218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57597 (local address 127.10.110.62:38165)
0926 16:42:36.886016 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.886028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.886072 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:42:36.887721 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.887747 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.888196 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.888581 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.889586 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.889592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.889605 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:36.889619 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.889677 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:36.889696 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.889897 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.890028 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.890231 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":356,"threads_started":1}
I20260926 16:42:36.891424 32672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae: Bootstrap starting.
I20260926 16:42:36.894536 32672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:36.894570 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:36.895493 32672 log.cc:824] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:36.898384 32672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae: No bootstrap required, opened a new log
I20260926 16:42:36.900982 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:36.898553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60663 (local address 127.10.110.62:38165)
0926 16:42:36.898730 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:36.898736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:36.899153 (+ 417us) server_negotiation.cc:407] Connection header received
0926 16:42:36.899291 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:36.899296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:36.899357 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:36.899463 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:36.900341 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:36.900347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:36.900355 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:36.900374 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:36.900428 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:36.900449 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:36.900572 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:36.900620 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 16:42:36.900691 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:36.906517 32672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638a2fbd40b4b67b103ab356ab635ae" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38165 } }
I20260926 16:42:36.906904 32672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:36.906977 32672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7638a2fbd40b4b67b103ab356ab635ae, State: Initialized, Role: FOLLOWER
I20260926 16:42:36.907476 32672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638a2fbd40b4b67b103ab356ab635ae" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38165 } }
I20260926 16:42:36.907716 32672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:36.907884 32672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:36.908078 32672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:36.910331 32672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638a2fbd40b4b67b103ab356ab635ae" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38165 } }
I20260926 16:42:36.910822 32672 leader_election.cc:302] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7638a2fbd40b4b67b103ab356ab635ae; no voters:
I20260926 16:42:36.911386 32672 leader_election.cc:288] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:36.911518 32682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:36.912155 32682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [term 1 LEADER]: Becoming Leader. State: Replica: 7638a2fbd40b4b67b103ab356ab635ae, State: Running, Role: LEADER
I20260926 16:42:36.912649 32682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638a2fbd40b4b67b103ab356ab635ae" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38165 } }
I20260926 16:42:36.913483 32672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:36.916028 32684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7638a2fbd40b4b67b103ab356ab635ae. Latest consensus state: current_term: 1 leader_uuid: "7638a2fbd40b4b67b103ab356ab635ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638a2fbd40b4b67b103ab356ab635ae" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38165 } } }
I20260926 16:42:36.915632 32683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7638a2fbd40b4b67b103ab356ab635ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7638a2fbd40b4b67b103ab356ab635ae" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 38165 } } }
I20260926 16:42:36.916548 32683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:36.916548 32684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:36.917609 32692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:36.922888 32692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:36.931686 32692 catalog_manager.cc:1347] Generated new cluster ID: 7cbfb5e1450a4b47b16f67c9a9325c9c
I20260926 16:42:36.931816 32692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:36.956660 32692 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:36.957108 32692 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:36.965324 32692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7638a2fbd40b4b67b103ab356ab635ae: Generated new TSK 0
I20260926 16:42:36.966226 32692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:36.971483 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32605
W20260926 16:42:36.972801 32667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57597
W20260926 16:42:36.973557 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:36.973693 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32605
2026-09-26T16:42:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1786 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:37 dist-test-slave-whjp krb5kdc[32703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:37 dist-test-slave-whjp krb5kdc[32703](info): set up 2 sockets
Sep 26 16:42:37 dist-test-slave-whjp krb5kdc[32703](info): commencing operation
krb5kdc: starting...
W20260926 16:42:37.993259 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:38 dist-test-slave-whjp krb5kdc[32703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:38.247473 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:35917
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:38.592957 32716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:38.593381 32716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:38.593529 32716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:38.603296 32716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:38.603415 32716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:38.603474 32716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:38.603555 32716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:38.603613 32716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:38.614110 32716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:38.615420 32716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:38.616761 32716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:38.628265 32722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.628243 32724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:38.628347 32721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:38.628995 32716 server_base.cc:1034] running on GCE node
I20260926 16:42:38.629769 32716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:38.631202 32716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:38.632398 32716 hybrid_clock.cc:648] HybridClock initialized: now 1790440958632349 us; error 38 us; skew 500 ppm
I20260926 16:42:38.632746 32716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:38 dist-test-slave-whjp krb5kdc[32703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:38.635766 32716 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:38.637274 32716 webserver.cc:460] Webserver started at http://127.10.110.62:38707/ using document root <none> and password file <none>
I20260926 16:42:38.637948 32716 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:38.638064 32716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:38.638356 32716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:38.641427 32716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "135f119a35bb47f8953a48e9cca64739"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-whjp"
server_key: "dadc886c071e1de7b0d2e4ba5c12cce4"
server_key_iv: "9857aa7066af2ba7921e1110011af186"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.642190 32716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "135f119a35bb47f8953a48e9cca64739"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-whjp"
server_key: "dadc886c071e1de7b0d2e4ba5c12cce4"
server_key_iv: "9857aa7066af2ba7921e1110011af186"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.647490 32716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260926 16:42:38.651189 32731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:38.652596 32716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:38.652755 32716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "135f119a35bb47f8953a48e9cca64739"
format_stamp: "Formatted at 2026-09-26 16:42:38 on dist-test-slave-whjp"
server_key: "dadc886c071e1de7b0d2e4ba5c12cce4"
server_key_iv: "9857aa7066af2ba7921e1110011af186"
server_key_version: "encryptionkey@0"
I20260926 16:42:38.652940 32716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:38.683197 32716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:38.687610 32716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:38.687909 32716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:38.705440 32716 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:35917
I20260926 16:42:38.705464 314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:35917 every 8 connection(s)
I20260926 16:42:38.706799 32716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:38.710889 315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:38.713691 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 32716
I20260926 16:42:38.713879 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:38.714449 10680 external_mini_cluster.cc:1428] Setting key f0f6a2462d3437cd9af8ce907638e6ce
I20260926 16:42:38.723989 315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739: Bootstrap starting.
I20260926 16:42:38.727965 315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739: Neither blocks nor log segments found. Creating new log.
Sep 26 16:42:38 dist-test-slave-whjp krb5kdc[32703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:38.729214 315 log.cc:824] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:38.732963 315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739: No bootstrap required, opened a new log
I20260926 16:42:38.738504 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.716579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41543 (local address 127.10.110.62:35917)
0926 16:42:38.717424 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.717437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.717480 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:42:38.719226 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.719251 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.719668 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.720070 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.721538 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.722963 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.724354 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.725405 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.725605 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.729756 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.729774 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.729800 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.729868 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.733785 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.734578 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.734588 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.734596 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.734815 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.735138 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.735142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.735145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.735935 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.736249 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.736589 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.737097 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":361,"threads_started":1}
I20260926 16:42:38.742455 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:38.743044 315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f119a35bb47f8953a48e9cca64739" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35917 } }
I20260926 16:42:38.743395 315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:38.743492 315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135f119a35bb47f8953a48e9cca64739, State: Initialized, Role: FOLLOWER
I20260926 16:42:38.744096 315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f119a35bb47f8953a48e9cca64739" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35917 } }
I20260926 16:42:38.744313 315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:38.744447 315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:38.744603 315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:38.746958 315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f119a35bb47f8953a48e9cca64739" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35917 } }
I20260926 16:42:38.747553 315 leader_election.cc:302] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135f119a35bb47f8953a48e9cca64739; no voters:
I20260926 16:42:38.748245 315 leader_election.cc:288] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:38.748409 325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:38.749876 325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [term 1 LEADER]: Becoming Leader. State: Replica: 135f119a35bb47f8953a48e9cca64739, State: Running, Role: LEADER
I20260926 16:42:38.750440 325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f119a35bb47f8953a48e9cca64739" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35917 } }
I20260926 16:42:38.750670 315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:38.753819 327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135f119a35bb47f8953a48e9cca64739. Latest consensus state: current_term: 1 leader_uuid: "135f119a35bb47f8953a48e9cca64739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f119a35bb47f8953a48e9cca64739" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35917 } } }
I20260926 16:42:38.754211 327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:38.754699 326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135f119a35bb47f8953a48e9cca64739" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135f119a35bb47f8953a48e9cca64739" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 35917 } } }
I20260926 16:42:38.754988 326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:38.755158 331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:38.763628 331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:38.764549 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:38.746814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52411 (local address 127.10.110.62:35917)
0926 16:42:38.747019 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:38.747026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:38.747441 (+ 415us) server_negotiation.cc:407] Connection header received
0926 16:42:38.747574 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:38.747580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:38.747660 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:38.747759 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:38.749639 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.750802 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.752593 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:38.753342 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:38.753525 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:38.756681 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:38.756688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:38.756692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:38.756730 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:38.760534 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.761410 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.761416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.761419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.761574 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:38.762028 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:38.762033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:38.762037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:38.762478 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:38.763770 (+ 1292us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:38.763937 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 16:42:38.764168 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:42:38.774082 331 catalog_manager.cc:1347] Generated new cluster ID: b249b7a35edd4bd5843270408d08adff
I20260926 16:42:38.774220 331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:38.793253 331 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:38.794749 331 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:38.804778 331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135f119a35bb47f8953a48e9cca64739: Generated new TSK 0
I20260926 16:42:38.805748 331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:38.821146 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 32716
I20260926 16:42:38.822367 310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:38.916654 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 32716
2026-09-26T16:42:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:39 dist-test-slave-whjp krb5kdc[347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:39 dist-test-slave-whjp krb5kdc[347](info): set up 2 sockets
Sep 26 16:42:39 dist-test-slave-whjp krb5kdc[347](info): commencing operation
krb5kdc: starting...
W20260926 16:42:39.968921 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:40 dist-test-slave-whjp krb5kdc[347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:40.232751 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39543
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:44261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:40.572705 360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:40.573022 360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:40.573143 360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:40.580706 360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:40.580835 360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:40.580891 360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:40.580977 360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:40.581029 360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:40.592094 360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:44261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:40.593490 360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:40.594820 360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:40.606832 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:40.606755 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:40.606809 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:40.607626 360 server_base.cc:1034] running on GCE node
I20260926 16:42:40.608448 360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:40.609861 360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:40.611099 360 hybrid_clock.cc:648] HybridClock initialized: now 1790440960611046 us; error 39 us; skew 500 ppm
I20260926 16:42:40.611456 360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:40 dist-test-slave-whjp krb5kdc[347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440960, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:40.619289 360 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:40.621073 360 webserver.cc:460] Webserver started at http://127.10.110.62:43971/ using document root <none> and password file <none>
I20260926 16:42:40.621814 360 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:40.621937 360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:40.622238 360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:40.625450 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "ed3fcf05a1204f7682c30f245eed10bb"
format_stamp: "Formatted at 2026-09-26 16:42:40 on dist-test-slave-whjp"
server_key: "292ce85f1db3836dbd30b8e3f3769737"
server_key_iv: "7379370c6eed7fe18d8fe8cd36720c53"
server_key_version: "encryptionkey@0"
I20260926 16:42:40.626312 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "ed3fcf05a1204f7682c30f245eed10bb"
format_stamp: "Formatted at 2026-09-26 16:42:40 on dist-test-slave-whjp"
server_key: "292ce85f1db3836dbd30b8e3f3769737"
server_key_iv: "7379370c6eed7fe18d8fe8cd36720c53"
server_key_version: "encryptionkey@0"
I20260926 16:42:40.632436 360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:42:40.636541 375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:40.638139 360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:40.638342 360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "ed3fcf05a1204f7682c30f245eed10bb"
format_stamp: "Formatted at 2026-09-26 16:42:40 on dist-test-slave-whjp"
server_key: "292ce85f1db3836dbd30b8e3f3769737"
server_key_iv: "7379370c6eed7fe18d8fe8cd36720c53"
server_key_version: "encryptionkey@0"
I20260926 16:42:40.638568 360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:40.660719 360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:40.665169 360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:40.665448 360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:40.683187 360 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:39543
I20260926 16:42:40.683214 427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:39543 every 8 connection(s)
I20260926 16:42:40.684592 360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:40.687808 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 360
I20260926 16:42:40.688017 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:40.688426 10680 external_mini_cluster.cc:1428] Setting key 0306c2753799a947971a92c9d95cbd1d
I20260926 16:42:40.688732 428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:40.702905 428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb: Bootstrap starting.
I20260926 16:42:40.702993 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:40.690174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57103 (local address 127.10.110.62:39543)
0926 16:42:40.690984 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:40.690996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:40.691038 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:42:40.693568 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:40.693592 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:40.694027 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:40.694405 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:40.695076 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:40.696493 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:40.698254 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:40.698998 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:40.699154 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:40.700607 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:40.700615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:40.700635 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:40.701224 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:40.701306 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:40.701335 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:40.701425 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:40.701691 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 16:42:40.701981 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":333,"threads_started":1}
I20260926 16:42:40.706559 428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:40.706902 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:40.707810 428 log.cc:824] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:40.711516 428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb: No bootstrap required, opened a new log
I20260926 16:42:40.718809 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:40.711637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49767 (local address 127.10.110.62:39543)
0926 16:42:40.711856 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:40.711863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:40.712454 (+ 591us) server_negotiation.cc:407] Connection header received
0926 16:42:40.712674 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:40.712678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:40.712746 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:40.712840 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:40.713509 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:40.714273 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:40.716193 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:40.716728 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:40.716867 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:40.718060 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:40.718065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:40.718069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:40.718094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:40.718140 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:40.718157 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:40.718235 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:40.718370 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 16:42:40.718452 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:42:40.721204 428 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3fcf05a1204f7682c30f245eed10bb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39543 } }
I20260926 16:42:40.721604 428 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:40.721698 428 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3fcf05a1204f7682c30f245eed10bb, State: Initialized, Role: FOLLOWER
I20260926 16:42:40.722259 428 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3fcf05a1204f7682c30f245eed10bb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39543 } }
I20260926 16:42:40.722482 428 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:40.722604 428 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:40.722762 428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:40.725075 428 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3fcf05a1204f7682c30f245eed10bb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39543 } }
I20260926 16:42:40.725606 428 leader_election.cc:302] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed3fcf05a1204f7682c30f245eed10bb; no voters:
I20260926 16:42:40.726364 428 leader_election.cc:288] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:40.726414 438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:40.727633 438 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [term 1 LEADER]: Becoming Leader. State: Replica: ed3fcf05a1204f7682c30f245eed10bb, State: Running, Role: LEADER
I20260926 16:42:40.728196 438 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3fcf05a1204f7682c30f245eed10bb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39543 } }
I20260926 16:42:40.728557 428 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:40.730733 440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed3fcf05a1204f7682c30f245eed10bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3fcf05a1204f7682c30f245eed10bb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39543 } } }
I20260926 16:42:40.731092 440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:40.731181 439 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed3fcf05a1204f7682c30f245eed10bb. Latest consensus state: current_term: 1 leader_uuid: "ed3fcf05a1204f7682c30f245eed10bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed3fcf05a1204f7682c30f245eed10bb" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39543 } } }
I20260926 16:42:40.731515 439 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:40.734264 446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:40.739480 446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:40.748880 446 catalog_manager.cc:1347] Generated new cluster ID: d60ae510daf646aab3de6934aa453a3f
I20260926 16:42:40.749011 446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:40.767473 446 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:40.768939 446 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:40.776082 446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed3fcf05a1204f7682c30f245eed10bb: Generated new TSK 0
I20260926 16:42:40.777345 446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:40.795069 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 360
W20260926 16:42:40.796509 423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57103
W20260926 16:42:40.797180 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 360: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:40.797341 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 360
2026-09-26T16:42:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1879 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:40 dist-test-slave-whjp krb5kdc[459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:40 dist-test-slave-whjp krb5kdc[459](info): set up 2 sockets
Sep 26 16:42:40 dist-test-slave-whjp krb5kdc[459](info): commencing operation
krb5kdc: starting...
W20260926 16:42:41.824968 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:41 dist-test-slave-whjp krb5kdc[459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:42.082873 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:37139
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:46697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:37139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:42.436880 472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:42.437179 472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:42.437291 472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:42.444761 472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:42.444890 472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:42.444942 472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:42.445003 472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:42.445045 472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:42.455159 472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:46697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:37139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:37139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:42.456530 472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:42.457829 472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:42.469786 477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:42.469820 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:42.469749 480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:42.470839 472 server_base.cc:1034] running on GCE node
I20260926 16:42:42.471700 472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:42.473376 472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:42.474622 472 hybrid_clock.cc:648] HybridClock initialized: now 1790440962474534 us; error 77 us; skew 500 ppm
I20260926 16:42:42.475021 472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:42 dist-test-slave-whjp krb5kdc[459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440962, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:42.478355 472 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:42.479978 472 webserver.cc:460] Webserver started at http://127.10.110.62:44055/ using document root <none> and password file <none>
I20260926 16:42:42.480742 472 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:42.480899 472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:42.481211 472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:42.484371 472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "ed42942e9ca74c84a86aca0c41e2b42b"
format_stamp: "Formatted at 2026-09-26 16:42:42 on dist-test-slave-whjp"
server_key: "688a3611cdaac38d13ad729bc1ae2c49"
server_key_iv: "7fe4ade2ca1764679a9785e82561ccdb"
server_key_version: "encryptionkey@0"
I20260926 16:42:42.485154 472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "ed42942e9ca74c84a86aca0c41e2b42b"
format_stamp: "Formatted at 2026-09-26 16:42:42 on dist-test-slave-whjp"
server_key: "688a3611cdaac38d13ad729bc1ae2c49"
server_key_iv: "7fe4ade2ca1764679a9785e82561ccdb"
server_key_version: "encryptionkey@0"
I20260926 16:42:42.491046 472 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:42:42.494930 487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:42.496652 472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:42.496847 472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "ed42942e9ca74c84a86aca0c41e2b42b"
format_stamp: "Formatted at 2026-09-26 16:42:42 on dist-test-slave-whjp"
server_key: "688a3611cdaac38d13ad729bc1ae2c49"
server_key_iv: "7fe4ade2ca1764679a9785e82561ccdb"
server_key_version: "encryptionkey@0"
I20260926 16:42:42.497123 472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:42.503223 472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:42.507795 472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:42.508140 472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:42.525771 472 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:37139
I20260926 16:42:42.525794 538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:37139 every 8 connection(s)
I20260926 16:42:42.527206 472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:42.527726 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 472
I20260926 16:42:42.527907 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:42.528442 10680 external_mini_cluster.cc:1428] Setting key 42a01c3be780e9a7398758b1eb840663
I20260926 16:42:42.531567 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:42.536435 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:42.530267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42215 (local address 127.10.110.62:37139)
0926 16:42:42.531033 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:42.531046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:42.531089 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:42:42.532883 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:42.532908 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:42.533280 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:42.533654 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:42.534708 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:42.534715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:42.534728 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:42.534743 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:42.534818 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:42.534839 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:42.535064 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:42.535227 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 16:42:42.535416 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":335,"threads_started":1}
I20260926 16:42:42.540495 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:42.543337 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b: Bootstrap starting.
I20260926 16:42:42.547096 539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:42.547382 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:42.544860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.10.110.62:37139)
0926 16:42:42.545043 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:42.545050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:42.545439 (+ 389us) server_negotiation.cc:407] Connection header received
0926 16:42:42.545607 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:42.545611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:42.545662 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:42.545751 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:42.546745 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:42.546750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:42.546755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:42:42.546791 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:42.546841 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:42:42.546862 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:42.546981 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:42.547054 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 16:42:42.547120 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:42.548550 539 log.cc:824] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:42.551512 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b: No bootstrap required, opened a new log
I20260926 16:42:42.559063 539 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 37139 } }
I20260926 16:42:42.559444 539 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:42:42.559517 539 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed42942e9ca74c84a86aca0c41e2b42b, State: Initialized, Role: FOLLOWER
I20260926 16:42:42.560087 539 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 37139 } }
I20260926 16:42:42.560297 539 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:42:42.560393 539 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:42:42.560516 539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:42.562649 539 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 37139 } }
I20260926 16:42:42.563143 539 leader_election.cc:302] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed42942e9ca74c84a86aca0c41e2b42b; no voters:
I20260926 16:42:42.563699 539 leader_election.cc:288] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:42:42.563867 549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:42.564882 549 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [term 1 LEADER]: Becoming Leader. State: Replica: ed42942e9ca74c84a86aca0c41e2b42b, State: Running, Role: LEADER
I20260926 16:42:42.565428 549 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 37139 } }
I20260926 16:42:42.565863 539 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:42.568364 551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed42942e9ca74c84a86aca0c41e2b42b. Latest consensus state: current_term: 1 leader_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 37139 } } }
I20260926 16:42:42.568771 551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:42.568329 550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed42942e9ca74c84a86aca0c41e2b42b" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 37139 } } }
I20260926 16:42:42.569130 550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:42.571666 557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:42.577293 557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:42.586570 557 catalog_manager.cc:1347] Generated new cluster ID: 4d97d6220c744697a52d12e3be129f69
I20260926 16:42:42.586700 557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:42.599450 557 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:42.600052 557 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:42.606122 557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed42942e9ca74c84a86aca0c41e2b42b: Generated new TSK 0
I20260926 16:42:42.607149 557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:42.614293 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 472
W20260926 16:42:42.615370 534 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42215
W20260926 16:42:42.616071 10680 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 472: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:42:42.616210 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 472
2026-09-26T16:42:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1818 ms)
[----------] 18 tests from AuthTokenIssuingTest (34343 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:42 dist-test-slave-whjp krb5kdc[570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:42 dist-test-slave-whjp krb5kdc[570](info): set up 2 sockets
Sep 26 16:42:42 dist-test-slave-whjp krb5kdc[570](info): commencing operation
krb5kdc: starting...
W20260926 16:42:43.665148 10680 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:43 dist-test-slave-whjp krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:43.928202 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:45251
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:45251,127.10.110.61:39031,127.10.110.60:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:44.261435 583 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:42:44.261798 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:44.261897 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:44.262018 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:44.269466 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:44.269574 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:44.269635 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:44.269742 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:44.269824 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:44.280110 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:45251,127.10.110.61:39031,127.10.110.60:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:44.281507 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:44.282891 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:44.295076 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:44.295130 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:44.295109 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:44.296000 583 server_base.cc:1034] running on GCE node
I20260926 16:42:44.296890 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:44.298405 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:44.299654 583 hybrid_clock.cc:648] HybridClock initialized: now 1790440964299568 us; error 66 us; skew 500 ppm
I20260926 16:42:44.300037 583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:44 dist-test-slave-whjp krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:44.303277 583 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:44.304807 583 webserver.cc:460] Webserver started at http://127.10.110.62:40125/ using document root <none> and password file <none>
I20260926 16:42:44.305490 583 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:44.305584 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:44.305846 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:44.308929 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "c84d68740bca46ba8467d500433240e1"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-whjp"
server_key: "463e58b4b5333d5c1b685622927a00e8"
server_key_iv: "75446b5f99460bc7a348efb80bfa1b51"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.309693 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "c84d68740bca46ba8467d500433240e1"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-whjp"
server_key: "463e58b4b5333d5c1b685622927a00e8"
server_key_iv: "75446b5f99460bc7a348efb80bfa1b51"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.314989 583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:44.318517 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:44.320015 583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 16:42:44.320176 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "c84d68740bca46ba8467d500433240e1"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-whjp"
server_key: "463e58b4b5333d5c1b685622927a00e8"
server_key_iv: "75446b5f99460bc7a348efb80bfa1b51"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.320374 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:44.334086 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:44.338245 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:44.338472 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:44.354818 583 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:45251
I20260926 16:42:44.354841 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:45251 every 8 connection(s)
I20260926 16:42:44.356227 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:44.360370 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:44.363080 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 583
I20260926 16:42:44.363256 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:44.363967 10680 external_mini_cluster.cc:1428] Setting key 6c14729e9f19177631427c08b8502ac2
I20260926 16:42:44.367591 650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:44 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.61@KRBTEST.COM" created.
I20260926 16:42:44.410358 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.10.110.62:45251)
0926 16:42:44.378100 (+ 5399us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:44.378110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:44.378153 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:42:44.378358 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:44.378367 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:44.378739 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:44.380150 (+ 1411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:44.381158 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.382965 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.385188 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.386304 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.386574 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.400617 (+ 14043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:44.400670 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:44.400697 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:44.400771 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:44.405109 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.405944 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.405954 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.405958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.406162 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.406610 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.406615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.406617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.407365 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:44.407586 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:44.408379 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 16:42:44.408726 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":5192,"thread_start_us":4996,"threads_started":1}
I20260926 16:42:44.410375 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.371907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:45251 (local address 127.0.0.1:43844)
0926 16:42:44.373357 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.373421 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:44.376255 (+ 2834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:44.379026 (+ 2771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:44.379040 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:44.379922 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:44.380879 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:44.380916 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.383172 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.383179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:44.384930 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:44.384954 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.386622 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.386629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:44.386856 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.387601 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:44.387627 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:44.390310 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:44.405292 (+ 14982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:44.405306 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:44.405321 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:44.405785 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:44.406287 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:44.406292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:44.406296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:44.406499 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:44.407511 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:44.407521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:44.407781 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 16:42:44.408095 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 16:42:44.408582 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":35,"spinlock_wait_cycles":20864,"thread_start_us":836,"threads_started":1}
I20260926 16:42:44.415233 650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:44.416550 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.416033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50432)
0926 16:42:44.416243 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.416398 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 16:42:44.416826 602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
I20260926 16:42:44.418500 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.418092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50448)
0926 16:42:44.418250 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.418331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:42:44.418700 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:39031: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab.
Entry for principal kudu/127.10.110.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab.
I20260926 16:42:44.451716 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:39031
--webserver_interface=127.10.110.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab
--principal=kudu/127.10.110.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:45251,127.10.110.61:39031,127.10.110.60:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:44.468060 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } attempt: 1
I20260926 16:42:44.469561 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.468991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50454)
0926 16:42:44.469230 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.469378 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 16:42:44.470806 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.470285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50466)
0926 16:42:44.470506 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.470611 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:42:44.471081 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:39031: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
I20260926 16:42:44.539463 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } attempt: 2
I20260926 16:42:44.540766 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.540230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50474)
0926 16:42:44.540458 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.540588 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 16:42:44.541868 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.541491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50486)
0926 16:42:44.541640 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.541721 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 16:42:44.542061 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:39031: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
I20260926 16:42:44.633317 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } attempt: 3
I20260926 16:42:44.634452 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.634037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50492)
0926 16:42:44.634233 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.634321 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 16:42:44.635411 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.635088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50504)
0926 16:42:44.635210 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.635279 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 16:42:44.635592 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:39031: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
I20260926 16:42:44.778869 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } attempt: 4
I20260926 16:42:44.780251 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.779683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50514)
0926 16:42:44.780006 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.780106 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260926 16:42:44.781565 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.781111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50522)
0926 16:42:44.781287 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.781398 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:42:44.781761 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:39031: Network error: Client connection negotiation failed: client connection to 127.10.110.61:39031: connect: Connection refused (error 111)
W20260926 16:42:44.797232 657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:42:44.797569 657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:44.797786 657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:44.797886 657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:44.805584 657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:44.805691 657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:44.805754 657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:44.805824 657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:44.805905 657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:44.816390 657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:45251,127.10.110.61:39031,127.10.110.60:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab
--principal=kudu/127.10.110.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:39031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.110.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:44.817731 657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:44.819113 657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:44.831053 665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:44.831103 663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:44.831107 662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:44.831646 657 server_base.cc:1034] running on GCE node
I20260926 16:42:44.832517 657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:44.833968 657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:44.835211 657 hybrid_clock.cc:648] HybridClock initialized: now 1790440964835128 us; error 51 us; skew 500 ppm
I20260926 16:42:44.835546 657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:44 dist-test-slave-whjp krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:44.838778 657 init.cc:377] Logged in from keytab as kudu/127.10.110.61@KRBTEST.COM (short username kudu)
I20260926 16:42:44.840377 657 webserver.cc:460] Webserver started at http://127.10.110.61:39051/ using document root <none> and password file <none>
I20260926 16:42:44.841089 657 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:44.841176 657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:44.841475 657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:44.844494 657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data/instance:
uuid: "028835e2ab424083931c7b2a2591f4d0"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-whjp"
server_key: "50b941c74969282b0c99aaa416cd5bd4"
server_key_iv: "0ad212cb56d0c1fc0f452e6686b92695"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.845279 657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal/instance:
uuid: "028835e2ab424083931c7b2a2591f4d0"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-whjp"
server_key: "50b941c74969282b0c99aaa416cd5bd4"
server_key_iv: "0ad212cb56d0c1fc0f452e6686b92695"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.850509 657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 16:42:44.854167 672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:44.855537 657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:42:44.855722 657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal
uuid: "028835e2ab424083931c7b2a2591f4d0"
format_stamp: "Formatted at 2026-09-26 16:42:44 on dist-test-slave-whjp"
server_key: "50b941c74969282b0c99aaa416cd5bd4"
server_key_iv: "0ad212cb56d0c1fc0f452e6686b92695"
server_key_version: "encryptionkey@0"
I20260926 16:42:44.855967 657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:44.876842 657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:44.881099 657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:44.881348 657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:44.898855 657 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.61:39031
I20260926 16:42:44.898874 723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.61:39031 every 8 connection(s)
I20260926 16:42:44.900235 657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
I20260926 16:42:44.904110 724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:44.908185 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 657
I20260926 16:42:44.908385 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-1/wal/instance
I20260926 16:42:44.908914 10680 external_mini_cluster.cc:1428] Setting key 7a936bed6343020126b3808e3ce771fe
I20260926 16:42:44.912578 724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:44 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.60@KRBTEST.COM; defaulting to no policy
I20260926 16:42:44.935777 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.915779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.10.110.62:45251)
0926 16:42:44.916303 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:44.916311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:44.917364 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 16:42:44.919941 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:44.919948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:44.920052 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:44.920166 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:44.921857 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.922621 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.924437 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.925039 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.925167 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.929017 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:44.929023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:44.929028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:44.929085 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:44.932491 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.933346 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.933352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.933356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.933517 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.933919 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.933924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.933927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.934337 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:44.934495 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:44.935272 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 16:42:44.935451 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":233,"threads_started":1}
I20260926 16:42:44.936955 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.916380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:45251 (local address 127.0.0.1:43850)
0926 16:42:44.917145 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.917227 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:44.919600 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:44.920443 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:44.920454 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:44.921014 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:44.921620 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:44.921640 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.922803 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.922807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:44.924254 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:44.924269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.925294 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.925303 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:44.925626 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.926320 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:44.926346 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:44.928761 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:44.932673 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:44.932680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:44.932696 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:44.933193 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:44.933618 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:44.933621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:44.933624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:44.933797 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:44.934480 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:44.934487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:44.934905 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 16:42:44.935154 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 16:42:44.935586 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":283,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.60@KRBTEST.COM" created.
I20260926 16:42:44.938758 724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:44 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:44.953310 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.939422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50526)
0926 16:42:44.939622 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.939638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:44.939736 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:44.940864 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:44.940868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:44.941288 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:44.941412 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:44.941420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.942717 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.942722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:44.943549 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:44.943560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.944377 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.944382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:44.944551 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.945164 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:44.945182 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:44.947419 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:44.950830 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:44.950835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:44.950840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:44.951210 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:44.951584 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:44.951587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:44.951590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:44.951720 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:44.952349 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:44.952353 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:44.952483 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 16:42:44.952599 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:42:44.952880 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:42:44.953351 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.939705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.10.110.61:39031)
0926 16:42:44.940365 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:44.940372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:44.940398 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:42:44.940443 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:44.940448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:44.940684 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:44.940851 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:44.941574 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.942592 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.943671 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:44.944245 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:44.944353 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:44.947659 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:44.947672 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:44.947686 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:44.947722 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:44.950673 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.951339 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.951346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.951349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.951474 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:44.951814 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:44.951819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:44.951821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:44.952220 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:44.952407 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:44.952749 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 16:42:44.952974 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":381,"threads_started":1}
I20260926 16:42:44.955972 724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:44.957183 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.956716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36080)
0926 16:42:44.956887 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.957002 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:42:44.957468 676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
I20260926 16:42:44.959157 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:44.958729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36090)
0926 16:42:44.958893 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:44.959026 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 16:42:44.959372 724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab.
Entry for principal kudu/127.10.110.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab.
I20260926 16:42:44.967461 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:46765
--webserver_interface=127.10.110.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab
--principal=kudu/127.10.110.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:45251,127.10.110.61:39031,127.10.110.60:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:45.008719 724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 1
I20260926 16:42:45.009963 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.009487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36096)
0926 16:42:45.009690 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.009805 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 16:42:45.010963 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.010612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36112)
0926 16:42:45.010775 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.010849 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:42:45.011193 724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
I20260926 16:42:45.079429 724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 2
I20260926 16:42:45.080839 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.080274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36126)
0926 16:42:45.080524 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.080642 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 16:42:45.081020 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } attempt: 5
I20260926 16:42:45.082192 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.081716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36134)
0926 16:42:45.081940 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.082033 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260926 16:42:45.082460 724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:45.095278 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.081716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.10.110.61:39031)
0926 16:42:45.081901 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.081908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.082050 (+ 142us) server_negotiation.cc:407] Connection header received
0926 16:42:45.082270 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.082276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.082388 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.082514 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.083326 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.084130 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.085385 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.085836 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.085970 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.089688 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.089694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.089698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.089734 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.092624 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.093356 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.093360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.093363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.093500 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.093870 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.093874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.093877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.094177 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.094313 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.094729 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.094944 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:42:45.095276 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.081686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50534)
0926 16:42:45.081950 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.081969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.082121 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.082559 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.082564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.082965 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.083169 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.083181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.084275 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.084282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.085218 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.085233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.085995 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.086002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.086217 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.086953 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.086975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.089432 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.092783 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.092790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.092794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.093225 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.093613 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.093619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.093622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.093764 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.094299 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.094304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.094474 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.094606 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.094916 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 16:42:45.096663 650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:45.097771 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.097315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36138)
0926 16:42:45.097495 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.097592 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 16:42:45.098712 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.098346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36146)
0926 16:42:45.098496 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.098580 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 16:42:45.098910 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
I20260926 16:42:45.150192 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 1
I20260926 16:42:45.151548 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.150952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36156)
0926 16:42:45.151213 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.151326 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 16:42:45.152683 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.152300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36170)
0926 16:42:45.152460 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.152550 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:42:45.152911 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
I20260926 16:42:45.173734 724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 3
I20260926 16:42:45.175093 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.174587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36178)
0926 16:42:45.174813 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.174902 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 16:42:45.176548 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.175950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36190)
0926 16:42:45.176284 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.176383 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260926 16:42:45.176864 724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
I20260926 16:42:45.221156 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 2
I20260926 16:42:45.222466 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.221878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36198)
0926 16:42:45.222137 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.222258 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 16:42:45.223683 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.223261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36204)
0926 16:42:45.223424 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.223521 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:42:45.223969 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
W20260926 16:42:45.314774 732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:42:45.315121 732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:45.315217 732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:45.315327 732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:42:45.320084 724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 4
I20260926 16:42:45.321297 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.320721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36208)
0926 16:42:45.320981 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.321098 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 16:42:45.322319 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.321955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36216)
0926 16:42:45.322106 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.322192 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:42:45.322546 724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
W20260926 16:42:45.323369 732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:45.323444 732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:45.323529 732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:45.323589 732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:45.323671 732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:45.330215 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 3
I20260926 16:42:45.331387 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.330873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36226)
0926 16:42:45.331073 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.331186 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 16:42:45.332427 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.332069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36242)
0926 16:42:45.332212 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.332304 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 16:42:45.332643 650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:46765: Network error: Client connection negotiation failed: client connection to 127.10.110.60:46765: connect: Connection refused (error 111)
I20260926 16:42:45.334729 732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:45251,127.10.110.61:39031,127.10.110.60:46765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab
--principal=kudu/127.10.110.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.110.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:45.336098 732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:45.337407 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:45.349356 738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.349407 740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:45.349356 737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:45.349804 732 server_base.cc:1034] running on GCE node
I20260926 16:42:45.350939 732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:45.352468 732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:45.353698 732 hybrid_clock.cc:648] HybridClock initialized: now 1790440965353640 us; error 40 us; skew 500 ppm
I20260926 16:42:45.354053 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:45.357141 732 init.cc:377] Logged in from keytab as kudu/127.10.110.60@KRBTEST.COM (short username kudu)
I20260926 16:42:45.358647 732 webserver.cc:460] Webserver started at http://127.10.110.60:40781/ using document root <none> and password file <none>
I20260926 16:42:45.359351 732 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:45.359445 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:45.359757 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:45.363188 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data/instance:
uuid: "6c17a35659314ca9a6107befd98da51e"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-whjp"
server_key: "925cf80e26636e9f9b253d684311036c"
server_key_iv: "f8a9a9ab85445059a3d0e8ef654d2278"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.364019 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal/instance:
uuid: "6c17a35659314ca9a6107befd98da51e"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-whjp"
server_key: "925cf80e26636e9f9b253d684311036c"
server_key_iv: "f8a9a9ab85445059a3d0e8ef654d2278"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.369390 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:42:45.373206 747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:45.374656 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 16:42:45.374847 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal
uuid: "6c17a35659314ca9a6107befd98da51e"
format_stamp: "Formatted at 2026-09-26 16:42:45 on dist-test-slave-whjp"
server_key: "925cf80e26636e9f9b253d684311036c"
server_key_iv: "f8a9a9ab85445059a3d0e8ef654d2278"
server_key_version: "encryptionkey@0"
I20260926 16:42:45.375090 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:45.397953 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:45.402165 732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:45.402419 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:45.419925 732 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.60:46765
I20260926 16:42:45.419960 798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.60:46765 every 8 connection(s)
I20260926 16:42:45.421286 732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
I20260926 16:42:45.423102 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 732
I20260926 16:42:45.423264 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440615325743-10680-0/minicluster-data/master-2/wal/instance
I20260926 16:42:45.423743 10680 external_mini_cluster.cc:1428] Setting key b876d2240c4944b5b10f1742693b2946
I20260926 16:42:45.425599 799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:45.434930 799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:45.448398 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.425516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.10.110.60:46765)
0926 16:42:45.426343 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.426357 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.426408 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:42:45.428582 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.428611 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.429139 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.429602 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.430977 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.432979 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.434523 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.435575 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.435786 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.440489 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.440500 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.440515 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.440565 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.443986 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.444723 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.444731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.444741 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.444933 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.445390 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.445393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.445395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.445993 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.446271 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.446642 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.447116 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260926 16:42:45.454106 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.437931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:45251 (local address 127.0.0.1:43858)
0926 16:42:45.438777 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.438824 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.438991 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.439394 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.439402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.440103 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.440399 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.440410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.441701 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.441707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.443385 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.443400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.444194 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.444199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.444324 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.444906 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.444924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.447484 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.451141 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.451149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.451161 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.451605 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.452190 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.452195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.452199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.452385 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.453013 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.453018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.453173 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.453338 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.453683 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":408,"threads_started":1}
I20260926 16:42:45.454110 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.437852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.10.110.62:45251)
0926 16:42:45.438381 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.438387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.438916 (+ 529us) server_negotiation.cc:407] Connection header received
0926 16:42:45.439121 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.439127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.439213 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.439305 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.440606 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.441495 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.443568 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.444016 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.444141 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.447738 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.447743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.447747 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.447791 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.451000 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.451867 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.451872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.451875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.452047 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.452546 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.452550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.452552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.452831 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.452966 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.453459 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.453683 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":254,"threads_started":1}
I20260926 16:42:45.455634 799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:45.467597 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.451851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.10.110.62:45251)
0926 16:42:45.452534 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.452538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.452556 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:42:45.452596 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.452600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.452654 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.452746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.453982 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.455092 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.456660 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.457278 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.457408 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.461805 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.461811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.461815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.461849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.464753 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.465514 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.465519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.465523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.465702 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.466074 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.466076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.466078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.466340 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.466523 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.466951 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.467181 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":315,"threads_started":1}
I20260926 16:42:45.469599 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.456350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:39031 (local address 127.0.0.1:50546)
0926 16:42:45.456597 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.456616 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.456713 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.456995 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.457000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.457459 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.457615 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.457623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.458905 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.458909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.459890 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.459902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.460849 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.460854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.461058 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.461591 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.461612 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.463758 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.466973 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.466978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.466981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.467471 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.467876 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.467881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.467884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.468060 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.468660 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.468665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.468859 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.469000 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.469284 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 16:42:45.469758 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.456524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.10.110.61:39031)
0926 16:42:45.456686 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.456691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.456705 (+ 14us) server_negotiation.cc:407] Connection header received
0926 16:42:45.456818 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.456822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.456881 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.456951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.457757 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.458736 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.460031 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.460501 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.460632 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.463961 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.463965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.463968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.464003 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.466842 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.467584 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.467588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.467590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.467747 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.468186 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.468190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.468192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.468528 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.468680 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.469101 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.469362 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 16:42:45.470911 799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440965, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:45.484248 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.471615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36252)
0926 16:42:45.471760 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.471778 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.471939 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.472211 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.472216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.472679 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.472861 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.472873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.474164 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.474168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.475348 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.475362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.476205 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.476210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.476345 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.476843 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.476860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.478867 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.481786 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.481792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.481796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.482293 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.482616 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.482621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.482625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.482784 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.483317 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.483320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.483443 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.483549 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.483874 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 16:42:45.484458 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.471713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.10.110.60:46765)
0926 16:42:45.471927 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.471933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.471949 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:45.472039 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.472042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.472095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.472168 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.472991 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.474030 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.475486 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.476038 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.476164 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.479050 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.479057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.479062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.479099 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.481660 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.482394 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.482398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.482400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.482512 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.482887 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.482890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.482892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.483207 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.483348 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.483660 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.484025 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":62}
I20260926 16:42:45.485929 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.469528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.10.110.61:39031)
0926 16:42:45.469950 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.469955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.470021 (+ 66us) server_negotiation.cc:407] Connection header received
0926 16:42:45.470257 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.470262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.470333 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.470426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.471932 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.472988 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.474553 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.475202 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.475334 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.479997 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.480002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.480005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.480040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.482989 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.483769 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.483774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.483778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.483967 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.484392 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.484397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.484401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.484755 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.484943 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.485399 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.485618 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":510,"threads_started":1}
I20260926 16:42:45.489711 799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: Bootstrap starting.
I20260926 16:42:45.490859 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:45.493084 799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:45.494393 799 log.cc:824] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:45.493434 625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c84d68740bca46ba8467d500433240e1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43866
I20260926 16:42:45.497547 799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: No bootstrap required, opened a new log
I20260926 16:42:45.508306 799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.508760 799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c17a35659314ca9a6107befd98da51e, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.509274 799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.509853 650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 4
I20260926 16:42:45.510336 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.510673 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.511650 799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
I20260926 16:42:45.527957 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.511180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.10.110.60:46765)
0926 16:42:45.511370 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.511376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.511396 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:42:45.511452 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.511458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.511530 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.511623 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.512490 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.513470 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.514819 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.515628 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.515762 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.520014 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.520022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.520028 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.520078 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.524344 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.525417 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.525424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.525428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.525612 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.526050 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.526056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.526059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.526528 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.526722 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.527308 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.527543 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:45.528290 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.510612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36256)
0926 16:42:45.510913 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.510934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.511064 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.511696 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.511702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.512165 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.512340 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.512350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.513602 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.513608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.514680 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.514694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.516035 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.516042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.516256 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.516881 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.516907 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.519741 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.524622 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.524628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.524639 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.525232 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.525750 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.525756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.525760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.525934 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.526703 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.526710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.526974 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.527234 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.527699 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 16:42:45.519496 732 master_runner.cc:186] Error getting master registration for 127.10.110.62:45251: OK, instance_id { permanent_uuid: "c84d68740bca46ba8467d500433240e1" instance_seqno: 1790440964350098 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:42:45.529588 732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:42:45.533735 819 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:45.533845 819 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:45.533998 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: Bootstrap starting.
I20260926 16:42:45.537143 650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:45.538172 650 log.cc:824] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:45.541617 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: No bootstrap required, opened a new log
I20260926 16:42:45.549619 650 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.550089 650 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84d68740bca46ba8467d500433240e1, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.550638 650 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.551687 822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.552147 822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.553000 650 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:45.553082 625 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:42:45.553262 625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.557229 625 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.561065 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c84d68740bca46ba8467d500433240e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6c17a35659314ca9a6107befd98da51e"
I20260926 16:42:45.561666 774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.565590 774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84d68740bca46ba8467d500433240e1 in term 1.
I20260926 16:42:45.566437 602 leader_election.cc:302] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c17a35659314ca9a6107befd98da51e, c84d68740bca46ba8467d500433240e1; no voters:
I20260926 16:42:45.567106 822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:45.572985 625 leader_election.cc:288] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [CANDIDATE]: Term 1 election: Requested vote from peers 028835e2ab424083931c7b2a2591f4d0 (127.10.110.61:39031), 6c17a35659314ca9a6107befd98da51e (127.10.110.60:46765)
I20260926 16:42:45.564320 583 master_runner.cc:186] Error getting master registration for 127.10.110.61:39031: OK, instance_id { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" instance_seqno: 1790440964893749 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:42:45.584820 822 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [term 1 LEADER]: Becoming Leader. State: Replica: c84d68740bca46ba8467d500433240e1, State: Running, Role: LEADER
I20260926 16:42:45.585687 583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:42:45.585615 822 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
W20260926 16:42:45.591539 604 tablet.cc:2369] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:45.591905 833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:45.595975 833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:45.597414 835 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84d68740bca46ba8467d500433240e1. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.598374 835 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:45.618410 842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:45.620787 842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:45.621730 724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } attempt: 5
Sep 26 16:42:45 dist-test-slave-whjp krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440964, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
I20260926 16:42:45.639739 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.622375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:46765 (local address 127.0.0.1:36264)
0926 16:42:45.622625 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:45.622654 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:45.622812 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:45.623483 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:45.623488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:45.624027 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:45.624220 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.624233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.625623 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.625628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.627086 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:45.627102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.628147 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.628152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:45.628369 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.629006 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:45.629038 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:45.631819 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:45.636026 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.636033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.636037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.636631 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.637375 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:45.637392 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:45.637396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:45.637596 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:45.638661 (+ 1065us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:45.638667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:45.638849 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 16:42:45.638987 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 16:42:45.639320 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 16:42:45.641898 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.622779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.10.110.60:46765)
0926 16:42:45.622955 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.622961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.622981 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:42:45.623063 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.623069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.623155 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.623267 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.624398 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.625491 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.627248 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.627995 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.628133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.632143 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.632150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.632155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.632202 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.635819 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.636868 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.636873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.636876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.637184 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.637797 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.637802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.637833 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.638444 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.640520 (+ 2076us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.640799 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.641234 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:42:45.653330 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: Bootstrap starting.
W20260926 16:42:45.654757 753 tablet.cc:2369] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:45.658123 724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:45.660213 724 log.cc:824] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:45.665836 805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.592014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.10.110.62:45251)
0926 16:42:45.592574 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.592585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.600565 (+ 7980us) server_negotiation.cc:407] Connection header received
0926 16:42:45.600881 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.600887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.600996 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.601095 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.612052 (+ 10957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.613277 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.615511 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.616303 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.616419 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.643213 (+ 26794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.643232 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.643239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.643293 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.651292 (+ 7999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.661173 (+ 9881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.661182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.661185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.661525 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.662359 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.662390 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.662393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.663197 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.663648 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.664787 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.665236 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:42:45.686470 724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: No bootstrap required, opened a new log
I20260926 16:42:45.687638 774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [term 1 FOLLOWER]: Refusing update from remote peer c84d68740bca46ba8467d500433240e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:45.689074 835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:45.704942 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.600930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.10.110.61:39031)
0926 16:42:45.601375 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.601382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.624909 (+ 23527us) server_negotiation.cc:407] Connection header received
0926 16:42:45.627311 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.627316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.627403 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.627468 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.639970 (+ 12502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.641039 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.672836 (+ 31797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.673481 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.673595 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.679865 (+ 6270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.679874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.679880 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.679955 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.689380 (+ 9425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.702306 (+ 12926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.702312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.702315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.702509 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.703053 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.703061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.703064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.703507 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.703807 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.704316 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.704598 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260926 16:42:45.708575 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84d68740bca46ba8467d500433240e1. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.708984 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.716066 847 mvcc.cc:205] Tried to move back new op lower bound from 7333646195412434944 to 7333646195046236160. Current Snapshot: MvccSnapshot[applied={T|T < 7333646195412434944}]
I20260926 16:42:45.723541 724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.724050 724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028835e2ab424083931c7b2a2591f4d0, State: Initialized, Role: FOLLOWER
I20260926 16:42:45.724668 724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } }
I20260926 16:42:45.726711 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.625058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.10.110.60:46765)
0926 16:42:45.642706 (+ 17648us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.642711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.667692 (+ 24981us) server_negotiation.cc:407] Connection header received
0926 16:42:45.668187 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.668194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.668310 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.668387 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.671540 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.672651 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.679521 (+ 6870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.680885 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.681052 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.719008 (+ 37956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:45.719017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:45.719022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:45.719073 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:45.722705 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.723753 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.723760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.723764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.723976 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:45.724476 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:45.724481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:45.724485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:45.724898 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:45.725131 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.725718 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.726103 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17422,"thread_start_us":49649,"threads_started":1}
I20260926 16:42:45.726351 852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.727885 852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.728497 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.728760 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.731938 835 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.732275 835 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:45.735247 842 catalog_manager.cc:1347] Generated new cluster ID: 7c3358b7c8f8449d87477b4db01e0656
I20260926 16:42:45.735368 842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:45.753909 724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:45.754004 699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c84d68740bca46ba8467d500433240e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "028835e2ab424083931c7b2a2591f4d0"
I20260926 16:42:45.754575 699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:45.757571 842 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:45.759410 842 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:45.759655 699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c84d68740bca46ba8467d500433240e1 in term 1.
I20260926 16:42:45.760586 698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [term 1 FOLLOWER]: Refusing update from remote peer c84d68740bca46ba8467d500433240e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:45.762270 853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:45.771327 842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1: Generated new TSK 0
I20260926 16:42:45.772715 842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:45.827929 852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84d68740bca46ba8467d500433240e1. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.828341 852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.828715 858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.829033 858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:45.830430 822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c84d68740bca46ba8467d500433240e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c84d68740bca46ba8467d500433240e1" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 45251 } } peers { permanent_uuid: "028835e2ab424083931c7b2a2591f4d0" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 39031 } } peers { permanent_uuid: "6c17a35659314ca9a6107befd98da51e" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 46765 } } }
I20260926 16:42:45.830765 822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84d68740bca46ba8467d500433240e1 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:45.874006 871 catalog_manager.cc:1259] Loaded cluster ID: 7c3358b7c8f8449d87477b4db01e0656
I20260926 16:42:45.874209 871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: loading cluster ID for follower catalog manager: success
I20260926 16:42:45.878515 871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: acquiring CA information for follower catalog manager: success
I20260926 16:42:45.882102 871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 028835e2ab424083931c7b2a2591f4d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:42:45.883059 10680 mini_kdc.cc:348] Time spent kdestroy: real 0.104s user 0.000s sys 0.024s
W20260926 16:42:45.929694 805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:45.919142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.10.110.62:45251)
0926 16:42:45.919318 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.919325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.926369 (+ 7044us) server_negotiation.cc:407] Connection header received
0926 16:42:45.927871 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.927878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.927990 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.928073 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.929341 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43884: BlockingRecv error: recv got EOF from 127.0.0.1:43884 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 16:42:45.933449 729 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:45.926946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.10.110.61:39031)
0926 16:42:45.927157 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.927163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.931174 (+ 4011us) server_negotiation.cc:407] Connection header received
0926 16:42:45.931972 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.931978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.932077 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.932152 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.933183 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50580: BlockingRecv error: recv got EOF from 127.0.0.1:50580 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 16:42:45.939030 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:45.936453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.10.110.60:46765)
0926 16:42:45.936624 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.936630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.937194 (+ 564us) server_negotiation.cc:407] Connection header received
0926 16:42:45.937444 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.937450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.937544 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.937615 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:45.938764 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36282: BlockingRecv error: recv got EOF from 127.0.0.1:36282 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:42:45.986388 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:45.970854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.10.110.61:39031)
0926 16:42:45.971066 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:45.971075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:45.978054 (+ 6979us) server_negotiation.cc:407] Connection header received
0926 16:42:45.978632 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:45.978639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:45.978864 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:45.979010 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:45.979770 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.980913 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.982955 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:45.983625 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:45.983731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:45.984488 (+ 757us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:45.985196 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:45.985320 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:45.985911 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 16:42:45.986029 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 16:42:46.032392 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.022438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.10.110.60:46765)
0926 16:42:46.022685 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.022692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.030015 (+ 7323us) server_negotiation.cc:407] Connection header received
0926 16:42:46.030637 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.030644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.030801 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.030893 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.032102 (+ 1209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36286: BlockingRecv error: recv got EOF from 127.0.0.1:36286 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:42:46.078553 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.063954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.10.110.61:39031)
0926 16:42:46.064198 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.064205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.071265 (+ 7060us) server_negotiation.cc:407] Connection header received
0926 16:42:46.071881 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.071887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.071992 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.072070 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.072820 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.073980 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.076106 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.076828 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.076940 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.077612 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.077796 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.077864 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.078102 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.078210 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 16:42:46.123064 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.113158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.10.110.60:46765)
0926 16:42:46.113398 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.113405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.120601 (+ 7196us) server_negotiation.cc:407] Connection header received
0926 16:42:46.121192 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.121199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.121321 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.121415 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.122824 (+ 1409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36296: BlockingRecv error: recv got EOF from 127.0.0.1:36296 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:42:46.184705 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.166750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.10.110.61:39031)
0926 16:42:46.166958 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.166965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.174030 (+ 7065us) server_negotiation.cc:407] Connection header received
0926 16:42:46.174656 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.174663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.174775 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.174853 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.176178 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.177425 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.182109 (+ 4684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.182794 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.182917 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.183707 (+ 790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.183919 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.184004 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.184258 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.184356 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 16:42:46.228327 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.218712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.10.110.60:46765)
0926 16:42:46.218896 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.218902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.225920 (+ 7018us) server_negotiation.cc:407] Connection header received
0926 16:42:46.226456 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.226462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.226557 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.226651 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.228053 (+ 1402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36298: BlockingRecv error: recv got EOF from 127.0.0.1:36298 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:42:46.279088 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.273387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.10.110.61:39031)
0926 16:42:46.273593 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.273599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.274363 (+ 764us) server_negotiation.cc:407] Connection header received
0926 16:42:46.274615 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.274619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.274695 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.274771 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.275211 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.276031 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.277382 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.277853 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.277977 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.278369 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.278483 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.278561 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.278707 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.278797 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 16:42:46.287915 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.285231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.10.110.60:46765)
0926 16:42:46.285488 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.285495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.286550 (+ 1055us) server_negotiation.cc:407] Connection header received
0926 16:42:46.286772 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.286778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.286856 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.286943 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.287663 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36300: BlockingRecv error: recv got EOF from 127.0.0.1:36300 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 16:42:46.311802 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.305777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.10.110.61:39031)
0926 16:42:46.306008 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.306016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.306543 (+ 527us) server_negotiation.cc:407] Connection header received
0926 16:42:46.306756 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.306762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.306835 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.306929 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.307333 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.308244 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.309778 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.310279 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.310400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.310851 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.310976 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.311058 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.311221 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.311311 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 16:42:46.326135 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.321670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.10.110.60:46765)
0926 16:42:46.321916 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.321925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.324577 (+ 2652us) server_negotiation.cc:407] Connection header received
0926 16:42:46.324849 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.324856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.324947 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.325053 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.325921 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36304: BlockingRecv error: recv got EOF from 127.0.0.1:36304 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:42:46.352319 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.346345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.10.110.61:39031)
0926 16:42:46.346558 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.346566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.347099 (+ 533us) server_negotiation.cc:407] Connection header received
0926 16:42:46.347321 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.347326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.347396 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.347495 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.347948 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.348873 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.350352 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.350917 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.351071 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.351542 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.351662 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.351739 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.351922 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.352022 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 16:42:46.361078 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.358963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.10.110.60:46765)
0926 16:42:46.359183 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.359190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.359692 (+ 502us) server_negotiation.cc:407] Connection header received
0926 16:42:46.359916 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.359921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.360010 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.360100 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.360890 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36316: BlockingRecv error: recv got EOF from 127.0.0.1:36316 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:42:46.404136 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.397869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.10.110.61:39031)
0926 16:42:46.398122 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.398128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.398452 (+ 324us) server_negotiation.cc:407] Connection header received
0926 16:42:46.398713 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.398717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.398790 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.398898 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.399373 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.400288 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.402229 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.402726 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.402834 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.403350 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.403458 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.403530 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.403708 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.403780 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 16:42:46.412128 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.409890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.10.110.60:46765)
0926 16:42:46.410136 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.410143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.410591 (+ 448us) server_negotiation.cc:407] Connection header received
0926 16:42:46.410767 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.410773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.410872 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.410988 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.411882 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36332: BlockingRecv error: recv got EOF from 127.0.0.1:36332 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 16:42:46.485270 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.480005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.10.110.61:39031)
0926 16:42:46.480237 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.480250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.480569 (+ 319us) server_negotiation.cc:407] Connection header received
0926 16:42:46.480805 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.480810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.480892 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.480972 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.481391 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.482179 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.483580 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.484074 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.484191 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.484591 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.484708 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.484775 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.484942 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.485019 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 16:42:46.494748 802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:46.492600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.10.110.60:46765)
0926 16:42:46.492773 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.492779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.493361 (+ 582us) server_negotiation.cc:407] Connection header received
0926 16:42:46.493582 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.493588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.493679 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.493766 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:46.494522 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36346: BlockingRecv error: recv got EOF from 127.0.0.1:36346 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 16:42:46.535921 819 catalog_manager.cc:1259] Loaded cluster ID: 7c3358b7c8f8449d87477b4db01e0656
I20260926 16:42:46.536042 819 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: loading cluster ID for follower catalog manager: success
I20260926 16:42:46.538622 819 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: acquiring CA information for follower catalog manager: success
I20260926 16:42:46.540585 819 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c17a35659314ca9a6107befd98da51e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:42:46.631639 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.626491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.10.110.61:39031)
0926 16:42:46.626704 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.626711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.626830 (+ 119us) server_negotiation.cc:407] Connection header received
0926 16:42:46.627072 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.627077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.627148 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.627227 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.627691 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.628490 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.629888 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.630322 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.630438 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.630867 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.631013 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.631095 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.631255 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.631336 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:42:46.644060 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.637908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.10.110.60:46765)
0926 16:42:46.638091 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.638095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.638372 (+ 277us) server_negotiation.cc:407] Connection header received
0926 16:42:46.638535 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.638538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.638693 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.638806 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.639199 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.640007 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.641545 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.642025 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.642150 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.642771 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.643338 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.643505 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.643610 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.643745 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 16:42:46.658418 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.651432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.10.110.61:39031)
0926 16:42:46.651614 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.651620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.651902 (+ 282us) server_negotiation.cc:407] Connection header received
0926 16:42:46.652193 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.652198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.652286 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.652351 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.653172 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.654272 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.656231 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.656918 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.657012 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.657620 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.657767 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.657847 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.658026 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.658140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 16:42:46.659597 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.650731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.10.110.62:45251)
0926 16:42:46.651149 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.651154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.651553 (+ 399us) server_negotiation.cc:407] Connection header received
0926 16:42:46.652003 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.652009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.652257 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.652437 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.653036 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.654268 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.656042 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.656563 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.656696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.657994 (+ 1298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.658451 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.658552 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.659149 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.659242 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":230,"threads_started":1}
I20260926 16:42:46.661264 802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:46.652028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.10.110.60:46765)
0926 16:42:46.652218 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:46.652224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:46.653458 (+ 1234us) server_negotiation.cc:407] Connection header received
0926 16:42:46.653660 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:46.653665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:46.653769 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:46.653845 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:46.655651 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.656712 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.658616 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:46.659305 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:46.659493 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:46.660528 (+ 1035us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:46.660643 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:46.660735 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:46.660985 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 16:42:46.661067 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 16:42:46.665458 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 583
I20260926 16:42:46.666647 645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:46.778307 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 583
W20260926 16:42:46.795393 749 connection.cc:535] server connection from 127.0.0.1:36256 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:42:46.795953 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 657
I20260926 16:42:46.796943 719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:46.905227 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 657
I20260926 16:42:46.922595 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 732
I20260926 16:42:46.923759 794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:47.031137 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 732
2026-09-26T16:42:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4420 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:42:47 dist-test-slave-whjp krb5kdc[982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:42:47 dist-test-slave-whjp krb5kdc[982](info): set up 2 sockets
Sep 26 16:42:47 dist-test-slave-whjp krb5kdc[982](info): commencing operation
krb5kdc: starting...
W20260926 16:42:48.061506 10680 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/test-user.keytab.
Sep 26 16:42:48 dist-test-slave-whjp krb5kdc[982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.110.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
Entry for principal kudu/127.10.110.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab.
I20260926 16:42:48.328579 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39711
--webserver_interface=127.10.110.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:39711,127.10.110.61:42545,127.10.110.60:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:42:48.675972 995 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:42:48.676332 995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:48.676429 995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:48.676551 995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:48.684005 995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:48.684100 995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:48.684158 995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:48.684223 995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:48.684299 995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:48.694449 995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:39711,127.10.110.61:42545,127.10.110.60:45799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.62.keytab
--principal=kudu/127.10.110.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.110.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:48.695785 995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:48.697114 995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:48.711058 1001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:48.711160 1000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:48.711133 1003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:48.711606 995 server_base.cc:1034] running on GCE node
I20260926 16:42:48.712636 995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:48.714167 995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:48.715405 995 hybrid_clock.cc:648] HybridClock initialized: now 1790440968715329 us; error 50 us; skew 500 ppm
I20260926 16:42:48.715731 995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:48 dist-test-slave-whjp krb5kdc[982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:48.718875 995 init.cc:377] Logged in from keytab as kudu/127.10.110.62@KRBTEST.COM (short username kudu)
I20260926 16:42:48.720429 995 webserver.cc:460] Webserver started at http://127.10.110.62:45453/ using document root <none> and password file <none>
I20260926 16:42:48.721125 995 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:48.721222 995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:48.721498 995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:48.724618 995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data/instance:
uuid: "53efb00f26be42f9b6673f834b2d9ef6"
format_stamp: "Formatted at 2026-09-26 16:42:48 on dist-test-slave-whjp"
server_key: "448f3392cec4e110f641248bc40b9ded"
server_key_iv: "a2123be0066f85ea94b781a1c6dfec69"
server_key_version: "encryptionkey@0"
I20260926 16:42:48.725397 995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal/instance:
uuid: "53efb00f26be42f9b6673f834b2d9ef6"
format_stamp: "Formatted at 2026-09-26 16:42:48 on dist-test-slave-whjp"
server_key: "448f3392cec4e110f641248bc40b9ded"
server_key_iv: "a2123be0066f85ea94b781a1c6dfec69"
server_key_version: "encryptionkey@0"
I20260926 16:42:48.730785 995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:42:48.734484 1010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:48.735978 995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:42:48.736163 995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal
uuid: "53efb00f26be42f9b6673f834b2d9ef6"
format_stamp: "Formatted at 2026-09-26 16:42:48 on dist-test-slave-whjp"
server_key: "448f3392cec4e110f641248bc40b9ded"
server_key_iv: "a2123be0066f85ea94b781a1c6dfec69"
server_key_version: "encryptionkey@0"
I20260926 16:42:48.736375 995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:48.756881 995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:48.761098 995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:48.761327 995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:48.778797 995 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.62:39711
I20260926 16:42:48.778820 1061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.62:39711 every 8 connection(s)
I20260926 16:42:48.780155 995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/data/info.pb
I20260926 16:42:48.784044 1062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:48.785059 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 995
I20260926 16:42:48.785243 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-0/wal/instance
I20260926 16:42:48.785718 10680 external_mini_cluster.cc:1428] Setting key 6ea519b8e4eecb3adc6b0ea1ee21b7c7
I20260926 16:42:48.790874 1062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:48 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.110.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.61@KRBTEST.COM" created.
I20260926 16:42:48.815299 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.794383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39711 (local address 127.0.0.1:37872)
0926 16:42:48.795278 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.795327 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:48.797569 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:48.798541 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:48.798549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:48.799310 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:48.799685 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:48.799709 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:48.801435 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:48.801440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:48.802635 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:48.802649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:48.803646 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:48.803654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:48.803875 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:48.804628 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:48.804653 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:48.807170 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:48.810851 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:48.810861 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:48.810872 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:48.811279 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:48.811696 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:48.811699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:48.811702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:48.811920 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:48.812713 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:48.812722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:48.812924 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 16:42:48.813159 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 16:42:48.813788 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":161,"thread_start_us":374,"threads_started":1}
I20260926 16:42:48.815318 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.794383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.10.110.62:39711)
0926 16:42:48.796173 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:48.796185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:48.796240 (+ 55us) server_negotiation.cc:407] Connection header received
0926 16:42:48.797998 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:48.798005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:48.798293 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:48.798497 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:48.799946 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:48.801252 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:48.802785 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:48.803387 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:48.803604 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:48.807444 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:48.807452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:48.807469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:48.807505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:48.810700 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:48.811412 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:48.811420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:48.811422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:48.811581 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:48.812028 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:48.812033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:48.812035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:48.812576 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:48.812779 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:48.813306 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 16:42:48.813663 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1564,"spinlock_wait_cycles":30464,"thread_start_us":1297,"threads_started":1}
I20260926 16:42:48.818349 1062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:48.819608 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.819173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59534)
0926 16:42:48.819350 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.819465 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 16:42:48.819970 1014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
I20260926 16:42:48.821480 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.821156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59540)
0926 16:42:48.821286 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.821355 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:42:48.821676 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:42545: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab.
Entry for principal kudu/127.10.110.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab.
I20260926 16:42:48.839332 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:42545
--webserver_interface=127.10.110.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab
--principal=kudu/127.10.110.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:39711,127.10.110.61:42545,127.10.110.60:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:48.871025 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } attempt: 1
I20260926 16:42:48.872200 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.871714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59548)
0926 16:42:48.871920 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.872070 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 16:42:48.873225 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.872841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59550)
0926 16:42:48.873027 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.873100 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 16:42:48.873416 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:42545: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
I20260926 16:42:48.941663 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } attempt: 2
I20260926 16:42:48.942800 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.942329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59556)
0926 16:42:48.942543 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.942668 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 16:42:48.943735 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:48.943402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59572)
0926 16:42:48.943528 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:48.943593 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 16:42:48.943930 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:42545: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
I20260926 16:42:49.035187 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } attempt: 3
I20260926 16:42:49.036422 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.035982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59586)
0926 16:42:49.036190 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.036283 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 16:42:49.037472 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.037086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59592)
0926 16:42:49.037247 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.037322 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 16:42:49.037667 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:42545: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
W20260926 16:42:49.166870 1069 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:42:49.167212 1069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:49.167309 1069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:49.167426 1069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:49.175063 1069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:49.175174 1069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:49.175231 1069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:49.175298 1069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:49.175380 1069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:49.180967 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } attempt: 4
I20260926 16:42:49.182206 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.181643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59598)
0926 16:42:49.181954 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.182045 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 16:42:49.183355 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.182851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59600)
0926 16:42:49.183081 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.183206 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:42:49.183555 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.61:42545: Network error: Client connection negotiation failed: client connection to 127.10.110.61:42545: connect: Connection refused (error 111)
I20260926 16:42:49.186395 1069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:39711,127.10.110.61:42545,127.10.110.60:45799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.61.keytab
--principal=kudu/127.10.110.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.61:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.110.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:49.187737 1069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:49.189062 1069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:49.201249 1077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:49.201316 1075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:49.201332 1074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:49.205534 1069 server_base.cc:1034] running on GCE node
I20260926 16:42:49.206374 1069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:49.207787 1069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:49.209049 1069 hybrid_clock.cc:648] HybridClock initialized: now 1790440969208973 us; error 47 us; skew 500 ppm
I20260926 16:42:49.209400 1069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:49.212620 1069 init.cc:377] Logged in from keytab as kudu/127.10.110.61@KRBTEST.COM (short username kudu)
I20260926 16:42:49.214150 1069 webserver.cc:460] Webserver started at http://127.10.110.61:45529/ using document root <none> and password file <none>
I20260926 16:42:49.214885 1069 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:49.215133 1069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:49.215487 1069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:49.218827 1069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data/instance:
uuid: "52a242875afe44a29d6cab3b51f4737b"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-whjp"
server_key: "0025271c5572246077c770056b42a9c9"
server_key_iv: "dfff3662120929a74ee35667e7a59547"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.219679 1069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal/instance:
uuid: "52a242875afe44a29d6cab3b51f4737b"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-whjp"
server_key: "0025271c5572246077c770056b42a9c9"
server_key_iv: "dfff3662120929a74ee35667e7a59547"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.225193 1069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:42:49.228844 1084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:49.230288 1069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 16:42:49.230453 1069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal
uuid: "52a242875afe44a29d6cab3b51f4737b"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-whjp"
server_key: "0025271c5572246077c770056b42a9c9"
server_key_iv: "dfff3662120929a74ee35667e7a59547"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.230711 1069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:49.246678 1069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:49.252048 1069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:49.252303 1069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:49.271031 1069 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.61:42545
I20260926 16:42:49.271057 1135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.61:42545 every 8 connection(s)
I20260926 16:42:49.272540 1069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/data/info.pb
I20260926 16:42:49.274124 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 1069
I20260926 16:42:49.274331 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-1/wal/instance
I20260926 16:42:49.274906 10680 external_mini_cluster.cc:1428] Setting key 2a0f0d367f580e4a5ded5a2f416883e3
I20260926 16:42:49.277397 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:49.286700 1136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.110.60@KRBTEST.COM; defaulting to no policy
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.110.60@KRBTEST.COM" created.
I20260926 16:42:49.307859 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.289604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.10.110.62:39711)
0926 16:42:49.289786 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.289790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.291227 (+ 1437us) server_negotiation.cc:407] Connection header received
0926 16:42:49.293295 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.293301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.293386 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.293475 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.295134 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.295811 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.297535 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.297970 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.298103 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.301687 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.301692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.301695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.301731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.304681 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.305516 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.305520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.305523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.305645 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.306063 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.306067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.306069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.306364 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.306505 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.307255 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.307499 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:42:49.308879 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.290228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39711 (local address 127.0.0.1:37878)
0926 16:42:49.291060 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.291100 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.292990 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.293771 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.293780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.294331 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.294926 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.294946 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.296011 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.296015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.297374 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.297392 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.298216 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.298223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.298465 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.299131 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.299159 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.301412 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.304868 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.304879 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.304895 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.305383 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.305755 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.305761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.305764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.305954 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.306490 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.306498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.306862 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.307132 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.307668 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":359,"threads_started":1}
I20260926 16:42:49.310999 1136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:49.325857 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.312120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59614)
0926 16:42:49.312308 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.312327 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.312455 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.313404 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.313409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.313887 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.314020 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.314029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.315429 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.315434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.316380 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.316394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.317203 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.317208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.317385 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.317956 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.317973 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.320283 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.323451 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.323456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.323460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.323823 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.324222 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.324228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.324232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.324368 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.324939 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.324944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.325082 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.325191 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.325464 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 16:42:49.325932 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.312188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.10.110.61:42545)
0926 16:42:49.312977 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.312983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.313009 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:42:49.313054 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.313060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.313258 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.313395 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.314191 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.315306 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.316507 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.317058 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.317175 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.320508 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.320524 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.320540 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.320571 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.323311 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.323993 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.324000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.324002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.324125 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.324466 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.324469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.324471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.324821 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.325003 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.325365 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.325584 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":494,"threads_started":1}
I20260926 16:42:49.328290 1136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:49.329479 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.329012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50260)
0926 16:42:49.329208 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.329322 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 16:42:49.329705 1087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.331211 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.330831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50264)
0926 16:42:49.330966 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.331070 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 16:42:49.331403 1136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.110.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab.
Entry for principal kudu/127.10.110.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab.
I20260926 16:42:49.336184 10680 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
/tmp/dist-test-taskiK5LtG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:45799
--webserver_interface=127.10.110.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab
--principal=kudu/127.10.110.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.110.20:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.110.62:39711,127.10.110.61:42545,127.10.110.60:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:42:49.380764 1136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 1
I20260926 16:42:49.382045 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.381504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50266)
0926 16:42:49.381740 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.381871 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 16:42:49.383021 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.382675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50282)
0926 16:42:49.382829 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.382908 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 16:42:49.383219 1136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.451452 1136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 2
I20260926 16:42:49.452790 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.452246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50296)
0926 16:42:49.452523 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.452624 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 16:42:49.453837 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.453478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50308)
0926 16:42:49.453628 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.453702 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 16:42:49.454061 1136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.482823 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } attempt: 5
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:49.496086 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.483661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.10.110.61:42545)
0926 16:42:49.483887 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.483893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.483914 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:42:49.484148 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.484151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.484233 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.484316 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.485174 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.485910 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.487063 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.487504 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.487621 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.490742 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.490748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.490752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.490788 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.493457 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.494168 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.494173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.494175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.494297 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.494663 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.494668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.494671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.494957 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.495095 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.495499 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.495739 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 16:42:49.496089 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.483596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59622)
0926 16:42:49.483810 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.483855 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.484009 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.484369 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.484374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.484805 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.485002 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.485012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.486061 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.486065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.486904 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.486916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.487659 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.487666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.487933 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.488529 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.488547 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.490527 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.493645 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.493651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.493653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.494044 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.494411 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.494414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.494417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.494555 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.495078 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.495083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.495245 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.495371 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.495703 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 16:42:49.497417 1062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:49.498680 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.498260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50316)
0926 16:42:49.498439 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.498532 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 16:42:49.499619 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.499287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50320)
0926 16:42:49.499427 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.499509 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 16:42:49.499814 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.545262 1136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 3
I20260926 16:42:49.546442 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.545945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50322)
0926 16:42:49.546166 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.546275 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:42:49.547796 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.547242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50328)
0926 16:42:49.547557 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.547655 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260926 16:42:49.548062 1136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.551110 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 1
I20260926 16:42:49.552207 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.551730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50344)
0926 16:42:49.551945 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.552040 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:42:49.553352 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.552936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50354)
0926 16:42:49.553131 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.553225 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 16:42:49.553576 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.621812 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 2
I20260926 16:42:49.623003 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.622519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50362)
0926 16:42:49.622741 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.622837 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 16:42:49.624218 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.623821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50364)
0926 16:42:49.624021 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.624097 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:42:49.624418 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
W20260926 16:42:49.678519 1143 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:42:49.678890 1143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:42:49.678990 1143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:42:49.679103 1143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:42:49.686491 1143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:42:49.686623 1143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:42:49.686707 1143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:42:49.686776 1143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:42:49.686860 1143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:42:49.691283 1136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 4
I20260926 16:42:49.692485 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.692011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50378)
0926 16:42:49.692238 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.692329 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:42:49.693547 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.693189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50390)
0926 16:42:49.693353 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.693424 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 16:42:49.693761 1136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.697662 1143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.110.20:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.110.62:39711,127.10.110.61:42545,127.10.110.60:45799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/krb5kdc/kudu_127.10.110.60.keytab
--principal=kudu/127.10.110.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.110.60:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.110.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:42:49.699249 1143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:42:49.700770 1143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:42:49.712409 1148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:49.712488 1151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:42:49.712438 1149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:42:49.713352 1143 server_base.cc:1034] running on GCE node
I20260926 16:42:49.714234 1143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:42:49.716121 1143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:42:49.717378 1143 hybrid_clock.cc:648] HybridClock initialized: now 1790440969717304 us; error 49 us; skew 500 ppm
I20260926 16:42:49.717741 1143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:42:49.721113 1143 init.cc:377] Logged in from keytab as kudu/127.10.110.60@KRBTEST.COM (short username kudu)
I20260926 16:42:49.723220 1143 webserver.cc:460] Webserver started at http://127.10.110.60:33583/ using document root <none> and password file <none>
I20260926 16:42:49.724012 1143 fs_manager.cc:359] Metadata directory not provided
I20260926 16:42:49.724128 1143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:42:49.724440 1143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:42:49.727514 1143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data/instance:
uuid: "ff9c655166ba417bbf835c7257e9dc86"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-whjp"
server_key: "9e6f1c3138898fbc5dadbf557a39f447"
server_key_iv: "d8a297a9a050726a1aad147f39c9f0f0"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.728400 1143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal/instance:
uuid: "ff9c655166ba417bbf835c7257e9dc86"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-whjp"
server_key: "9e6f1c3138898fbc5dadbf557a39f447"
server_key_iv: "d8a297a9a050726a1aad147f39c9f0f0"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.730660 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 3
I20260926 16:42:49.731935 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.731316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50394)
0926 16:42:49.731598 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.731699 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 16:42:49.733225 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.732749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50408)
0926 16:42:49.732978 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.733076 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 16:42:49.733423 1062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.110.60:45799: Network error: Client connection negotiation failed: client connection to 127.10.110.60:45799: connect: Connection refused (error 111)
I20260926 16:42:49.734542 1143 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:42:49.738354 1158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:42:49.739777 1143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 16:42:49.740001 1143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data,/tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal
uuid: "ff9c655166ba417bbf835c7257e9dc86"
format_stamp: "Formatted at 2026-09-26 16:42:49 on dist-test-slave-whjp"
server_key: "9e6f1c3138898fbc5dadbf557a39f447"
server_key_iv: "d8a297a9a050726a1aad147f39c9f0f0"
server_key_version: "encryptionkey@0"
I20260926 16:42:49.740245 1143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:42:49.758632 1143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:42:49.763232 1143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:42:49.763497 1143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:42:49.781200 1143 rpc_server.cc:307] RPC server started. Bound to: 127.10.110.60:45799
I20260926 16:42:49.781222 1209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.110.60:45799 every 8 connection(s)
I20260926 16:42:49.782532 1143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/data/info.pb
I20260926 16:42:49.786988 1210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:42:49.791726 10680 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu as pid 1143
I20260926 16:42:49.791950 10680 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiK5LtG/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440615325743-10680-0/minicluster-data/master-2/wal/instance
I20260926 16:42:49.792493 10680 external_mini_cluster.cc:1428] Setting key b445361b12a3a5967787957f5013de6d
I20260926 16:42:49.794417 1210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
I20260926 16:42:49.815630 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.797467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.10.110.62:39711)
0926 16:42:49.797732 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.797737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.798424 (+ 687us) server_negotiation.cc:407] Connection header received
0926 16:42:49.798916 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.798922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.799002 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.799107 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.800831 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.801930 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.804178 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.804797 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.804931 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.808914 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.808921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.808926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.808970 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.811737 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.812844 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.812850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.812852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.813005 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.813456 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.813462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.813465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.813844 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.813974 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.814953 (+ 979us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.815212 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 16:42:49.816823 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.797481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.62:39711 (local address 127.0.0.1:37886)
0926 16:42:49.798253 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.798313 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.798666 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.799200 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.799207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.800072 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.800641 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.800669 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.802121 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.802125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.804021 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.804035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.805054 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.805062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.805236 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.806012 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.806045 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.808693 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.811935 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.811949 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.811964 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.812660 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.813135 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.813140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.813144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.813325 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.813978 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.813986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.814444 (+ 458us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.814802 (+ 358us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.815316 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"spinlock_wait_cycles":17152,"thread_start_us":348,"threads_started":1}
I20260926 16:42:49.816841 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.794750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.10.110.60:45799)
0926 16:42:49.795764 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.795780 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.796191 (+ 411us) server_negotiation.cc:407] Connection header received
0926 16:42:49.798332 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.798355 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.798692 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.798912 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.800678 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.802331 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.804114 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.805001 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.805172 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.809619 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.809630 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.809644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.809678 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.813022 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.813884 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.813890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.813892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.814038 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.814386 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.814391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.814394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.814826 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.815110 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.815452 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.815692 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":455,"threads_started":1}
I20260926 16:42:49.818444 1210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.62@KRBTEST.COM
I20260926 16:42:49.833160 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.819381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.10.110.61:42545)
0926 16:42:49.819600 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.819606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.819851 (+ 245us) server_negotiation.cc:407] Connection header received
0926 16:42:49.820016 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.820022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.820107 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.820213 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.821075 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.821779 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.823096 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.823507 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.823601 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.827049 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.827057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.827062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.827110 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.829971 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.830916 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.830923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.830927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.831115 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.831621 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.831626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.831629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.832030 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.832199 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.832655 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.832893 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:42:49.833197 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.819402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.61:42545 (local address 127.0.0.1:59634)
0926 16:42:49.819734 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.819752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.819890 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.820268 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.820273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.820704 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.820916 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.820928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.821918 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.821921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.822953 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.822967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.823649 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.823655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.823889 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.824466 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.824489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.826808 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.830167 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.830188 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.830192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.830756 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.831256 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.831263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.831268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.831496 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.832187 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.832191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.832351 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.832535 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.832794 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260926 16:42:49.834296 1210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:42:49.836191 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.820017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.10.110.62:39711)
0926 16:42:49.820190 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.820196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.820212 (+ 16us) server_negotiation.cc:407] Connection header received
0926 16:42:49.820358 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.820363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.820428 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.820501 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.821792 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.822551 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.823961 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.824384 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.824476 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.829438 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.829447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.829452 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.829501 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.832686 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.833745 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.833755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.833757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.833923 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.834421 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.834425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.834428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.834834 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.835485 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.835630 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.835869 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.60@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.110.61@KRBTEST.COM
I20260926 16:42:49.847321 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.835127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50418)
0926 16:42:49.835365 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.835385 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.835496 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.835821 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.835863 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.836329 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.836485 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.836494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.837558 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.837562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.838501 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.838513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.839184 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.839189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.839376 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.839995 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.840013 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.842243 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.845213 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.845218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.845223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.845558 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.845863 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.845868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.845872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.845989 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.846537 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.846541 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.846673 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.846803 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.847052 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 16:42:49.847539 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.835097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.10.110.60:45799)
0926 16:42:49.835330 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.835336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.835449 (+ 113us) server_negotiation.cc:407] Connection header received
0926 16:42:49.835601 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.835607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.835682 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.835781 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.836617 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.837421 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.838631 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.839053 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.839160 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.842428 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.842434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.842437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.842467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.845090 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.845662 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.845667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.845669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.845770 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.846096 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.846101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.846105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.846394 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.846555 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.846935 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.847217 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":107}
I20260926 16:42:49.853710 1210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: Bootstrap starting.
I20260926 16:42:49.854236 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.838141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.10.110.61:42545)
0926 16:42:49.838299 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.838305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.838388 (+ 83us) server_negotiation.cc:407] Connection header received
0926 16:42:49.838559 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.838565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.838630 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.838711 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.840101 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.840865 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.842337 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.842758 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.842871 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.848041 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.848049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.848053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.848100 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.851441 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.852253 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.852260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.852263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.852428 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.852816 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.852820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.852823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.853184 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.853459 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.853674 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.853912 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 16:42:49.857429 1210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:49.858667 1210 log.cc:824] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:49.858937 10680 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:42:49.861747 1210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: No bootstrap required, opened a new log
I20260926 16:42:49.861536 1037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "53efb00f26be42f9b6673f834b2d9ef6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37890
I20260926 16:42:49.868957 1210 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:49.869355 1210 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9c655166ba417bbf835c7257e9dc86, State: Initialized, Role: FOLLOWER
I20260926 16:42:49.869817 1210 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:49.870771 1216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:49.871158 1216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:49.872015 1210 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:42:49.881805 1227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:42:49.881950 1227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:42:49.879478 1143 master_runner.cc:186] Error getting master registration for 127.10.110.62:39711: OK, instance_id { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" instance_seqno: 1790440968774068 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:42:49.887688 1143 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:42:49.910629 1062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 4
Sep 26 16:42:49 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440968, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.62@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
I20260926 16:42:49.924999 1065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.911503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50422)
0926 16:42:49.911719 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.911740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.911890 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:49.912342 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:49.912346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:49.912788 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:49.912963 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.912974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.913997 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.914001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.914958 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:49.914972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.915736 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.915741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:49.915969 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.916546 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:49.916565 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:49.918745 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:49.922191 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.922197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.922202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.922601 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.922976 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:49.922981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:49.922985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:49.923135 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:49.923954 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:49.923965 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:49.924144 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 16:42:49.924294 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 16:42:49.924595 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:42:49.925030 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.911569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.10.110.60:45799)
0926 16:42:49.911879 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.911885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.911907 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:42:49.912105 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.912112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.912193 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.912279 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.913115 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.913848 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.915096 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.915590 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.915708 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.918968 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.918976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.918981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.919024 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.922037 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.922724 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.922730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.922734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.922861 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:49.923265 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:49.923272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:49.923275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:49.923773 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:49.924008 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:49.924414 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 16:42:49.924670 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 16:42:49.931077 1062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6: Bootstrap starting.
I20260926 16:42:49.935148 1062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:49.936527 1062 log.cc:824] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:49.939606 1062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6: No bootstrap required, opened a new log
I20260926 16:42:49.945890 1062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:49.946338 1062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53efb00f26be42f9b6673f834b2d9ef6, State: Initialized, Role: FOLLOWER
I20260926 16:42:49.946786 1062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:49.947728 1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:49.948205 1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:49.948932 1062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:49.949048 1037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:42:49.949297 1037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:49.952908 1037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:49.954612 1037 leader_election.cc:288] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [CANDIDATE]: Term 1 election: Requested vote from peers 52a242875afe44a29d6cab3b51f4737b (127.10.110.61:42545), ff9c655166ba417bbf835c7257e9dc86 (127.10.110.60:45799)
I20260926 16:42:49.956588 1185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53efb00f26be42f9b6673f834b2d9ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff9c655166ba417bbf835c7257e9dc86"
I20260926 16:42:49.957145 1185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:49.962242 1185 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53efb00f26be42f9b6673f834b2d9ef6 in term 1.
I20260926 16:42:49.963546 1013 leader_election.cc:302] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53efb00f26be42f9b6673f834b2d9ef6, ff9c655166ba417bbf835c7257e9dc86; no voters:
I20260926 16:42:49.964212 1230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:42:49.964864 1230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [term 1 LEADER]: Becoming Leader. State: Replica: 53efb00f26be42f9b6673f834b2d9ef6, State: Running, Role: LEADER
I20260926 16:42:49.965492 1230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:49.989890 1236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53efb00f26be42f9b6673f834b2d9ef6. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:49.990373 1236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:49.992944 1136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } attempt: 5
I20260926 16:42:50.002619 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.980693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.10.110.60:45799)
0926 16:42:49.980854 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.980860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.982893 (+ 2033us) server_negotiation.cc:407] Connection header received
0926 16:42:49.983224 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.983230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.983325 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.983401 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.986005 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.987072 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.990080 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.990787 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.990900 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.995153 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.995161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.995167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.995218 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:49.999169 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.000050 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.000056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.000059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.000235 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.000705 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.000711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.000714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.001064 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.001337 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.001921 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.002195 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:42:50.008189 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.980090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.10.110.61:42545)
0926 16:42:49.980275 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.980281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.984534 (+ 4253us) server_negotiation.cc:407] Connection header received
0926 16:42:49.984680 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.984685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.984770 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.984885 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.986763 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.987963 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.990934 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.991710 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.991821 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:49.997562 (+ 5741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:49.997569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:49.997574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:49.997621 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:50.001294 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.002654 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.002660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.002664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.002830 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.003730 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.003736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.003739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.004178 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.004416 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.007547 (+ 3131us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.007805 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 16:42:50 dist-test-slave-whjp krb5kdc[982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.110.61@KRBTEST.COM for kudu/127.10.110.60@KRBTEST.COM
W20260926 16:42:50.014339 1016 tablet.cc:2369] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:42:50.015775 1164 tablet.cc:2369] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:42:50.022178 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.993552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.110.60:45799 (local address 127.0.0.1:50434)
0926 16:42:49.993965 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 16:42:49.993983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:42:49.994109 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:42:50.003466 (+ 9357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:42:50.003471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:42:50.004119 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:42:50.004288 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.004297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.005959 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.005966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.007448 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:42:50.007467 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.009141 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.009149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:42:50.009388 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.010055 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:42:50.010073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:42:50.012696 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:42:50.018694 (+ 5998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.018700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.018705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.019315 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.020089 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:42:50.020094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:42:50.020099 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:42:50.020278 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:42:50.021062 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:42:50.021068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:42:50.021241 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 16:42:50.021391 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 16:42:50.021729 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260926 16:42:50.023528 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.994328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.10.110.60:45799)
0926 16:42:50.003165 (+ 8837us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.003171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.003189 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:42:50.003249 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.003255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.003331 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.003426 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.004448 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.005762 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.007671 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.008894 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.009728 (+ 834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.012973 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:50.012982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:50.012987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:50.013054 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:50.018487 (+ 5433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.019690 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.019696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.019701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.019926 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.020405 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.020411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.020414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.020838 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.020979 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.022829 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.023122 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8716,"threads_started":1}
I20260926 16:42:50.023571 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:49.979990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.10.110.62:39711)
0926 16:42:49.980201 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:49.980209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:49.981216 (+ 1007us) server_negotiation.cc:407] Connection header received
0926 16:42:49.981432 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:49.981438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:49.981515 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:49.981625 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:49.988088 (+ 6463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:49.989138 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:49.999747 (+ 10609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:50.000474 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:50.000584 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:50.013898 (+ 13314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:42:50.013906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:42:50.013912 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:42:50.013966 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:42:50.017820 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.019357 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.019367 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.019371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.019707 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:42:50.020475 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:42:50.020481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:42:50.020505 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:42:50.021107 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:42:50.021572 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:50.022518 (+ 946us) server_negotiation.cc:299] Negotiation successful
0926 16:42:50.022960 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:42:50.028880 1245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:42:50.036960 1136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: Bootstrap starting.
I20260926 16:42:50.043639 1245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:42:50.044883 1136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: Neither blocks nor log segments found. Creating new log.
I20260926 16:42:50.047243 1136 log.cc:824] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:42:50.064029 1136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: No bootstrap required, opened a new log
I20260926 16:42:50.068430 1185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [term 1 FOLLOWER]: Refusing update from remote peer 53efb00f26be42f9b6673f834b2d9ef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:50.070248 1236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:42:50.092263 1216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53efb00f26be42f9b6673f834b2d9ef6. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.092602 1216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:50.092873 1253 mvcc.cc:205] Tried to move back new op lower bound from 7333646213384949760 to 7333646212984528896. Current Snapshot: MvccSnapshot[applied={T|T < 7333646213384949760}]
I20260926 16:42:50.096578 1216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.096948 1216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:50.108343 1236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.108732 1236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:42:50.116322 1255 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:42:50.117375 1245 catalog_manager.cc:1347] Generated new cluster ID: 40b5fa41f4e5481baf53c65e92aef492
I20260926 16:42:50.118234 1245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:42:50.115794 1136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:50.118933 1136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a242875afe44a29d6cab3b51f4737b, State: Initialized, Role: FOLLOWER
I20260926 16:42:50.119721 1136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } }
I20260926 16:42:50.166600 1245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:42:50.125403 995 master_runner.cc:186] Error getting master registration for 127.10.110.61:42545: OK, instance_id { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" instance_seqno: 1790440969265704 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:42:50.173384 995 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:42:50.173566 1245 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:42:50.179651 1260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.181660 1260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:50.182861 1136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:42:50.183329 1110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:42:50.187407 1110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [term 1 FOLLOWER]: Refusing update from remote peer 53efb00f26be42f9b6673f834b2d9ef6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:42:50.187958 1111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53efb00f26be42f9b6673f834b2d9ef6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "52a242875afe44a29d6cab3b51f4737b"
I20260926 16:42:50.191665 1111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53efb00f26be42f9b6673f834b2d9ef6 in term 1.
I20260926 16:42:50.193737 1236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260926 16:42:50.221725 1245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6: Generated new TSK 0
I20260926 16:42:50.223016 1245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:42:50.241065 1260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53efb00f26be42f9b6673f834b2d9ef6. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.241575 1260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:50.242146 1264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.242575 1264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:42:50.245190 1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53efb00f26be42f9b6673f834b2d9ef6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53efb00f26be42f9b6673f834b2d9ef6" member_type: VOTER last_known_addr { host: "127.10.110.62" port: 39711 } } peers { permanent_uuid: "52a242875afe44a29d6cab3b51f4737b" member_type: VOTER last_known_addr { host: "127.10.110.61" port: 42545 } } peers { permanent_uuid: "ff9c655166ba417bbf835c7257e9dc86" member_type: VOTER last_known_addr { host: "127.10.110.60" port: 45799 } } }
I20260926 16:42:50.245625 1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53efb00f26be42f9b6673f834b2d9ef6 [sys.catalog]: This master's current role is: LEADER
W20260926 16:42:50.366720 1066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.342138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.10.110.62:39711)
0926 16:42:50.342368 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.342374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.363187 (+ 20813us) server_negotiation.cc:407] Connection header received
0926 16:42:50.365010 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.365018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.365126 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.365216 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.366444 (+ 1228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37906: BlockingRecv error: recv got EOF from 127.0.0.1:37906 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:42:50.366905 1285 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 16:42:50.375538 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.360351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.10.110.61:42545)
0926 16:42:50.360579 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.360586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.373728 (+ 13142us) server_negotiation.cc:407] Connection header received
0926 16:42:50.373909 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.373914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.374017 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.374113 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.375273 (+ 1160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59668: BlockingRecv error: recv got EOF from 127.0.0.1:59668 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 16:42:50.376641 1246 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.348614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.10.110.60:45799)
0926 16:42:50.349014 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.349020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.372277 (+ 23257us) server_negotiation.cc:407] Connection header received
0926 16:42:50.375186 (+ 2909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.375192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.375280 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.375358 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.376381 (+ 1023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50446: BlockingRecv error: recv got EOF from 127.0.0.1:50446 (error 108)
Metrics: {"server-negotiator.queue_time_us":202}
W20260926 16:42:50.386802 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.384156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.10.110.61:42545)
0926 16:42:50.384354 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.384360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.385208 (+ 848us) server_negotiation.cc:407] Connection header received
0926 16:42:50.385430 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.385436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.385567 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.385678 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.386505 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59672: BlockingRecv error: recv got EOF from 127.0.0.1:59672 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:42:50.398136 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.391322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.10.110.61:42545)
0926 16:42:50.391502 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.391508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.396478 (+ 4970us) server_negotiation.cc:407] Connection header received
0926 16:42:50.396809 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.396815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.396918 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.397050 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.397895 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59682: BlockingRecv error: recv got EOF from 127.0.0.1:59682 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 16:42:50.405632 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.403399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.10.110.61:42545)
0926 16:42:50.403575 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.403581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.404160 (+ 579us) server_negotiation.cc:407] Connection header received
0926 16:42:50.404347 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.404353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.404478 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.404584 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.405382 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59684: BlockingRecv error: recv got EOF from 127.0.0.1:59684 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:42:50.420295 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.417952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.10.110.61:42545)
0926 16:42:50.418145 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.418151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.418728 (+ 577us) server_negotiation.cc:407] Connection header received
0926 16:42:50.418919 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.418924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.419031 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.419143 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.420069 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59686: BlockingRecv error: recv got EOF from 127.0.0.1:59686 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 16:42:50.433557 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.431663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.10.110.61:42545)
0926 16:42:50.431907 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.431913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.432110 (+ 197us) server_negotiation.cc:407] Connection header received
0926 16:42:50.432283 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.432287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.432359 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.432440 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.433274 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59700: BlockingRecv error: recv got EOF from 127.0.0.1:59700 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 16:42:50.454648 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.452664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.10.110.61:42545)
0926 16:42:50.452875 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.452880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.453195 (+ 315us) server_negotiation.cc:407] Connection header received
0926 16:42:50.453383 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.453388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.453463 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.453565 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.454416 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59712: BlockingRecv error: recv got EOF from 127.0.0.1:59712 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 16:42:50.492511 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.490523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.10.110.61:42545)
0926 16:42:50.490739 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.490745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.491112 (+ 367us) server_negotiation.cc:407] Connection header received
0926 16:42:50.491272 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.491277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.491375 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.491506 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.492231 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59724: BlockingRecv error: recv got EOF from 127.0.0.1:59724 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 16:42:50.562760 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.560828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.10.110.61:42545)
0926 16:42:50.561041 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.561047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.561318 (+ 271us) server_negotiation.cc:407] Connection header received
0926 16:42:50.561521 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.561525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.561613 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.561700 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.562547 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59734: BlockingRecv error: recv got EOF from 127.0.0.1:59734 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 16:42:50.699630 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.697721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.10.110.61:42545)
0926 16:42:50.697934 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.697950 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.698290 (+ 340us) server_negotiation.cc:407] Connection header received
0926 16:42:50.698494 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.698500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.698584 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.698672 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.699420 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59742: BlockingRecv error: recv got EOF from 127.0.0.1:59742 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:42:50.883916 1227 catalog_manager.cc:1259] Loaded cluster ID: 40b5fa41f4e5481baf53c65e92aef492
I20260926 16:42:50.884038 1227 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: loading cluster ID for follower catalog manager: success
I20260926 16:42:50.886543 1227 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: acquiring CA information for follower catalog manager: success
I20260926 16:42:50.888345 1227 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff9c655166ba417bbf835c7257e9dc86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:42:50.960834 1141 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:42:50.959235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.10.110.61:42545)
0926 16:42:50.959404 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:50.959409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:50.959644 (+ 235us) server_negotiation.cc:407] Connection header received
0926 16:42:50.959852 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:50.959856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:50.959934 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:50.960013 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:42:50.960663 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59748: BlockingRecv error: recv got EOF from 127.0.0.1:59748 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 16:42:51.371407 1285 catalog_manager.cc:1259] Loaded cluster ID: 40b5fa41f4e5481baf53c65e92aef492
I20260926 16:42:51.371536 1285 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: loading cluster ID for follower catalog manager: success
I20260926 16:42:51.374382 1285 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: acquiring CA information for follower catalog manager: success
I20260926 16:42:51.376425 1285 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 52a242875afe44a29d6cab3b51f4737b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:42:51.482460 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.476464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.10.110.61:42545)
0926 16:42:51.477269 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.477275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.477294 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:42:51.477366 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.477370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.477514 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.477635 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:51.478052 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.478834 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.480187 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.480648 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.480758 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.481243 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:51.481817 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:51.481949 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:51.482075 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 16:42:51.482160 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":468,"threads_started":1}
I20260926 16:42:51.495875 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.488908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.10.110.60:45799)
0926 16:42:51.489465 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.489471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.489786 (+ 315us) server_negotiation.cc:407] Connection header received
0926 16:42:51.490016 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.490020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.490205 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.490358 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:51.490804 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.491516 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.493498 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.493926 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.494044 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.494683 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:51.495203 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:51.495307 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:51.495428 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 16:42:51.495500 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
I20260926 16:42:51.521517 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.503079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.10.110.60:45799)
0926 16:42:51.503258 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.503264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.514744 (+ 11480us) server_negotiation.cc:407] Connection header received
0926 16:42:51.515027 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.515034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.515166 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.515268 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:51.516016 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.517043 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.519300 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.519932 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.520037 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.520727 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:51.520851 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:51.520908 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:51.521121 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 16:42:51.521205 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:42:51.523185 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.502841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.10.110.61:42545)
0926 16:42:51.503025 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.503032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.515478 (+ 12446us) server_negotiation.cc:407] Connection header received
0926 16:42:51.516192 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.516199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.516287 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.516348 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:51.517299 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.518305 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.520715 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.521382 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.521484 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.522361 (+ 877us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:51.522486 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:51.522546 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:51.522770 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 16:42:51.522854 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:42:51.523428 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:42:51.502814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.10.110.62:39711)
0926 16:42:51.515819 (+ 13005us) server_negotiation.cc:206] Beginning negotiation
0926 16:42:51.515823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:42:51.515869 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:42:51.515990 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:42:51.515995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:42:51.516230 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:42:51.517101 (+ 871us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:42:51.517276 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.518341 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.520609 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:42:51.521190 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:42:51.521792 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:42:51.521986 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:42:51.522433 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:42:51.523020 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0926 16:42:51.523069 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 16:42:51.523149 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12907,"thread_start_us":12808,"threads_started":1}
I20260926 16:42:51.533332 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 995
I20260926 16:42:51.534736 1057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:51.703331 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 995
W20260926 16:42:51.720131 1160 connection.cc:535] server connection from 127.0.0.1:50422 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:42:51.720809 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 1069
I20260926 16:42:51.721915 1131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:51.836112 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 1069
I20260926 16:42:51.851786 10680 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu pid 1143
I20260926 16:42:51.852989 1205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:42:51.960166 10680 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiK5LtG/build/asan/bin/kudu with pid 1143
2026-09-26T16:42:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4927 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9348 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356599 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST